JP7569748B2 - Storage - Google Patents

Description

The present invention relates to a storage facility and a method for storing items in the storage facility.

A technology has been proposed for automatically identifying food entering a storage compartment such as a refrigerator (Patent Document 1). This patent document states that "by capturing images of food entering the storage compartment at appropriate times, it is possible to more reliably obtain information about the food entering the storage compartment." There is also a known technology that makes it possible to automatically manage inventory in a storage compartment (Patent Document 2).

Patent Publication No. 2019-70476 Patent No. 6473523

With conventional technology, there is no fixed location for placing food inside the refrigerator, so there is room for improvement in the accuracy of recognizing food based on a photograph of the food.

The present invention was made in consideration of the above problems, and its purpose is to provide a storage facility and a method for storing items in the storage facility that can improve the accuracy of food recognition.

To solve the above problems, a storage facility according to the present invention includes a storage facility main body, a camera unit that photographs the interior of the storage facility main body, a notification unit that notifies a user of a specific storage unit among the multiple storage units of the storage facility main body, and a control unit that, when a target item is stored in the storage facility main body, photographs the target item with the camera unit and recognizes it through image recognition processing, selects a specific storage unit to store the recognized target item based on a specific table that manages the priority order of storage units to store items, and causes the notification unit to notify the user of the selected specific storage unit.

According to the present invention, it is possible to manage the storage section that stores the items, and to improve the accuracy of recognizing the items.

FIG. 1 is an overall configuration diagram of a refrigerator interior management system. FIG. 2 is an explanatory diagram showing an overview of a notification means. FIG. 2 is a perspective view showing a mounting configuration of the camera. 11 is an explanatory diagram showing the relationship between the fisheye camera and the mirror and the upper shelf. FIG. 1 is an example of a camera image. 1 is an example of an image obtained by converting an image taken by a fisheye camera into a planar image. 13 is a flowchart showing an in-warehouse management process. 13 is an example of a table for managing food storage locations. 1 is an example of a food list displayed on a user's mobile device. 13 is a flowchart showing a food ordering process. 13 is an example of an order screen displayed on a mobile terminal. 1 shows an example of a stock table. 13 is a flowchart showing an in-warehouse management process in the second embodiment. 13 is an example of an updated storage location management table. 13 is a flowchart showing an in-warehouse management process in a third embodiment. 13 is a flowchart showing an in-warehouse management process in a fourth embodiment. 13 is a flowchart showing a process for registering a new food item in the storage location management table. 13 is a flowchart illustrating a process in which a user inputs information regarding the success or failure of a food recognition result in a fifth embodiment.

Below, an embodiment of the present invention will be described with reference to the drawings. In this embodiment, as described below, among the storage sections in the storehouse, the locations where items are easily recognized are identified in advance, and the storage sections in which the items are to be placed are prioritized and managed in a specified table. Then, when a user attempts to store an item in the storehouse, the target item is photographed and subjected to image recognition processing, and the user is guided to the storage section in which the recognized item can be stored. Since items are stored in storage sections according to the type of item, the recognition accuracy by the image recognition processing improves the more the storehouse is used.

In the examples described below, a refrigerator will be used as an example of a storage unit. Food items will be used as an example of an item. However, this embodiment is not limited to storing food in a refrigerator, and can be widely applied to storing non-food items such as medicines in a refrigerator, and storing items such as parts on an item shelf.

This embodiment can also be expressed as follows. That is, a refrigerator having a control unit having a processor and a memory, and an openable/closable door, the refrigerator has an opening sensor that detects the opening degree of the door, a camera that can capture an image of the interior of the storage unit from information when the door is opened, and a reflector that reflects the interior of the storage unit to the camera when the door is opened. The control unit has a photographing unit that captures an image of the interior of the storage unit and the reflector as a camera image when the opening degree detected by the opening sensor exceeds a predetermined threshold value, a recognition unit that recognizes items from the camera image captured by the camera, a table control unit that sets a priority order for each item recognized by the recognition unit, a notification unit that notifies the user of the position to place food by light and sound based on the table control unit for each item recognized by the recognition unit, and a recognition result output unit that outputs information on the recognized items in text.

The system may have a table control unit that determines the priority of the placement of each item in the warehouse so that it is easy to recognize.

The first embodiment will be described with reference to Figures 1 to 12. The refrigerator 1 includes a control unit 10 and a refrigerator main body 20 as a "storage main body." The refrigerator system includes the refrigerator 1 and external devices 7, 8, and 9 that are connected to the refrigerator 1 via a communication network CN so as to be capable of two-way communication.

The mobile terminal 7 as an external device is a terminal used by the user of the refrigerator 1. The mobile terminal 7 displays a food list transmitted from the refrigerator 1 as described below. The external device 8 is a web server. The external device 9 is a computer. The web server 8 includes, for example, an online supermarket and a recipe site. The computer 9 is a computer for distributing machine learning models and the like.

The refrigerator body 20 includes a left door 21L and a right door 21R that are attached so as to be freely opened and closed, an upper door pocket 23U, a middle door pocket 23M, and a lower door pocket 23L that are attached to each of the doors 21L and 21R, an upper shelf 22U, a middle shelf 22M, and a lower shelf 22L within the refrigerator chamber that are arranged from the top side of the refrigerator body 20 to approximately the center in the height direction, a chilled compartment 24, an ice making compartment 25, a fresh freezing room 26, an upper vegetable compartment 27U, a lower vegetable compartment 27L, and a freezer compartment 28.

In the following explanation, unless otherwise specified, the left and right doors 21L, 21R may be abbreviated as doors 21, the upper door pocket 23U, the middle door pocket 23M, and the lower door pocket 23L as door pockets 23, and the upper shelf 22U, the middle shelf 22M, and the lower shelf 22L as shelves 22.

Opening sensors 40 that detect the opening degree of the left and right doors 21 are provided on the top of the refrigerator body 20, as described below. In addition, a camera 50 that takes pictures of the interior of the refrigerator is attached to the top of the refrigerator body 20, as described below.

The control unit 10 that controls the refrigerator 1 includes, for example, a processor 11, a storage device 12, a communication unit (I/F in the figure) 14 connected to the network CN, and an I/O interface (I/O in the figure) 13 connected to the opening sensor 40 and the camera 50. The storage device 12 includes a main storage device consisting of a volatile or non-volatile memory, and an auxiliary storage device consisting of a flash memory, a hard disk drive, or the like. Some or all of the computer programs and data stored in the storage device 12 can also be transmitted to the outside via the communications network CN. Conversely, computer programs and data can also be transmitted from an external computer 9, etc., to the storage device 12 via the communications network CN for storage.

A storage medium MM such as a flash memory or a hard disk drive can be connected to the control unit 10, and some or all of the computer programs and data can be transferred between the storage device 12 and the storage medium MM.

The storage device 12 stores specific computer programs that realize the image capture unit 121, image buffer 122, image conversion unit 123, recognition unit 124, table control unit 125, guidance unit 126, recognition result output unit 127, and interior control unit 128. The processor 11 executes these specific computer programs to realize each of the functional units 121 to 128.

In other words, processor 11 operates as a functional unit that provides a specified function by executing processing according to a specified computer program. For example, processor 11 functions as image conversion unit 123 by executing processing according to an image conversion program. The same applies to other computer programs. Furthermore, processor 11 also operates as a functional unit that provides each function of multiple processes executed by each computer program.

The photographing unit 121 acquires camera images from the camera 50 via the I/O interface 13, and stores the acquired camera images in the image buffer 122. The camera 50 as the "camera unit" is configured as a fisheye camera. The camera 50 has, for example, a fisheye or wide-angle lens 52 (see FIG. 4).

As described below, when the opening degree of the door 21 detected by the opening degree sensor 40 exceeds a predetermined threshold value Th1 and the door 21 stops opening, the photographing unit 121 photographs the interior of the refrigerator 1 with the camera 50 and stores the photographed image as a camera image in the image buffer 122. The interior is the inside of the refrigerator body 20.

The timing for capturing an image of the interior of the fridge may be when at least one of the left and right opening sensors 40 exceeds a predetermined threshold Th1 and when the door 21 stops opening. In other words, the camera 50 may capture an image of the interior of the fridge when either one of the doors 21 is open, not just when both the left and right doors 21L, 21R are open by more than the threshold Th1.

The image conversion unit 123 converts a camera image captured by a fisheye or wide-angle lens 52 into a planar image. Since publicly known techniques can be used to convert an image captured by a fisheye or wide-angle lens into a planar image, a description thereof will be omitted.

The recognition unit 124 includes a learning-based image recognition unit 1241, a rule-based image recognition unit 1242, and a storage location detection unit 1243. The learning-based image recognition unit 1241 and the rule-based image recognition unit 1242 each recognize food in the storage unit from the converted planar image. The learning-based image recognition unit 1241 includes, for example, a machine learning model such as a neural network that has been trained in advance, and when a planar image is input, it outputs a recognition result of the food (item) included in the planar image.

When a two-dimensional image is input, the rule-based image recognition unit 1242 recognizes the food inside the fridge using a rule base. The rule-based image recognition unit 1242 divides the input two-dimensional image into regions, labels the items in each region, and outputs the contents of the recognized labels in text. Publicly known technologies can be applied to the learning-based image recognition unit 1241 and the rule-based image recognition unit 1242.

The table control unit 125 controls the contents of the storage location management table T1 (see FIG. 8), which defines the priority of the location for each food item. The storage location management table T1, which serves as a "predetermined table," is provided inside the table control unit 125.

Here, the storage location or placement position refers to the place where food is stored. The place where food is stored refers to the storage section that stores food. In this embodiment, the upper door pocket 23U, the middle door pocket 23M, the lower door pocket 23L, the upper shelf 22U, the middle shelf 22M, the lower shelf 22L, the chilled compartment 24, the ice making compartment 25, the fresh freezing room 26, the upper vegetable compartment 27U, the lower vegetable compartment 27L, and the freezer compartment 28 are given as examples of storage sections. In other words, the storage section is a partitioned space provided within the refrigerator body 20.

The guidance unit 126 searches for and selects the food storage location recognized by the recognition unit 124 from the storage location management table T1, and activates the notification means 30 corresponding to the selected storage location. The notification means 30 as a "notification unit" is provided individually for each storage location. The notification means 30 for each storage location includes, for example, at least one of a speaker 31 or a light 32. The notification means 30 in this embodiment has both a speaker 31 and a light 32.

The speaker 31 notifies the user of the presence of the selected storage location by sound. The notification sound may be music stored in the refrigerator 1 when it is shipped from the factory, or music registered by the user after purchasing the refrigerator 1. The notification sound may be a voice message such as "Please put milk in the lower door pocket." The voice message may include the food recognition result by the recognition unit 124. The notification sound may be a combination of music and a voice message. The shape and number of the speakers 31 are not important.

The light 32 notifies the user of the presence of the selected storage location with light. The light 32 may be a light with multiple light-emitting elements arranged in a matrix, or may be one or multiple surface-emitting lights. Light-emitting elements that do not generate heat, such as LEDs (Light Emitting Diodes), may be used for the light 32. Elements that do not generate heat other than LEDs may also be used. In the case of a storage facility that is not required to be kept at low temperatures, such as the refrigerator 1, but is intended to be stored at room temperature, elements that emit light while generating heat may be used.

Figure 2 shows an example of the arrangement of the notification means 30. In Figure 2, an example of the arrangement of the notification means 30 is explained using the chilled compartment 24 and the vegetable compartment 27 as examples. In this embodiment, a flat speaker 31 and a light 32 consisting of point-like LED light-emitting elements arranged in a matrix are placed on the bottom surface of each storage area. The bottom surface is the flat surface on which food is placed.

Returning to FIG. 1, the placement location detection unit 1243 detects where the user has placed the food in the storage unit. The placement location detection unit 123 can detect where the food has been placed using known technology.

The recognition result output unit 127 compiles the types and quantities of food in the refrigerator from the recognition results from the machine learning model and the rule-based recognition results, and generates a food list 72 (see FIG. 9) for the refrigerator. The recognition result output unit 127 then instructs the communication unit 14 to send the generated food list 72 to a pre-configured mobile terminal 7. The mobile terminal 7 is running a refrigerator management application (not shown), and displays the food list received from the refrigerator 1 on the display unit 71 (see FIG. 9) of the mobile terminal 7.

The interior control unit 128 controls the motor and compressor (not shown) to control the temperature and humidity inside the refrigerator 1.

Figure 3 is a perspective view showing a partially enlarged view of the upper part of the refrigerator 1. Hinge parts 201 are attached to both the left and right sides of the top plate 200 at the top of the refrigerator body 20. The left and right doors 21 are supported by the hinge parts 201 so that they can be opened and closed freely. In this embodiment, the left and right doors 21 are so-called double-door doors, and an example is shown in which the left and right sides of the opening 202 of the refrigerator 1 can be opened and closed independently.

An opening sensor 40 that detects the opening degree (angle) of the door 21 is attached to the upper side of the left and right hinge parts 201. A camera 50 is attached to the top surface of the top plate 200, positioned halfway between the left and right hinge parts 201. A front end 51 of the camera 50 protrudes further forward than the opening 202 of the refrigerator 1, and is configured to capture an image of the interior of the refrigerator from above when the door 21 is open.

Figure 4 shows an example of how the camera 50 is attached. The camera 50 is attached to the top surface of the top plate 200 with its end (or end face) 51 protruding a predetermined amount from the opening 202 (the end of the top plate 200) toward the front of the refrigerator 1.

The camera 50 is configured as a fisheye camera, with a step formed in front of the camera 50 and a fisheye lens 52 provided on the underside of the step. The mirror 60 is provided so as to cover the fisheye lens 52 from the front of the camera 50.

That is, a mirror 60 is provided hanging down from the end 51 of the camera 50 that protrudes beyond the opening 202. The mirror 60 functions as a reflector that guides to the camera 50 at least the image of an item placed on the floor surface (flat surface) of the upper shelf 22U and the image of an item placed on the floor surface (flat surface) of the middle shelf 22M among the storage sections of the refrigerator body 20.

If the mirror 60 and the door 21 interfere with each other when the door 21 is opened or closed, the mirror 60 may be designed to be foldable or displaceable. In the illustrated example, the mirror 60 is attached to the end 51 of the camera 50, but this is not limiting and the mirror 60 may be installed further forward of the end 51 of the refrigerator 1.

Figure 5 shows an example of a camera image GC captured by camera 50. The camera image GC captured by fisheye lens 52 includes the interior of refrigerator 1 and pockets 23 on left and right doors 21 captured directly through lens 52, the top surfaces of upper shelf 22U, middle shelf 22M, and lower shelf 22L reflected in mirror 60, and food placed in chilled compartment 24, fresh freezing room 26, and vegetable compartment 27.

Even with a fisheye lens 52, an image taken directly from the lens 52 can only capture a portion of the items on the top surface of the top shelf 22U. In contrast, an image reflected by the mirror 60 can capture food placed on the top surface of the top shelf 22U all the way to the back of the fridge.

However, it is difficult to use the camera image GC taken with the fisheye lens 52 as is to have the learning-based image recognition unit 1241 or the rule-based image recognition unit 1242 of the recognition unit 124 recognize food. In other words, in the learning phase of the machine learning model, learning is performed with undistorted images, so it is difficult to recognize the distorted camera image GC taken with the fisheye or wide-angle lens 52 as is.

For this reason, the image conversion unit 123 converts the distorted image captured by the fisheye lens 52 into a planar image. Figure 6 shows the fisheye camera image GC converted into a planar image.

The image conversion unit 123 extracts an interior direct image (first image) 45 taken directly from the lens 52 and a mirror image (second image) 41 taken from the lens 52 via a mirror 60 from the camera image GC taken with the fisheye lens 52, and converts each of the interior direct image 45 and mirror image 41 into a planar image with the distortion of the lens 52 removed.

A known camera calibration technique may be used as a method for removing distortion. As one example, an image with a known pattern after distortion correction, such as a checkerboard, may be used to extract feature points from the images before and after distortion correction, and the parameters of the fisheye camera may be estimated based on the position coordinates of these feature points.

The direct interior image G10 is composed of a right door image G11, a front interior image G12, and a left door image G13, and is approximately the upper half of the camera image GC in FIG. 5. The mirror image G20 is an image of the interior of the camera image GC in FIG. 5 reflected by the lower mirror 60.

The image conversion unit 123 converts the right door image G11 of the fridge interior direct image G10 into a planar image to generate a right door unfolded image G32. The image conversion unit 123 also converts the fridge interior front image G12 into a planar image to generate a front unfolded image G33. The image conversion unit 123 also converts the left door image G34 into a planar image to generate a left door unfolded image G34.

The image conversion unit 123 also converts the mirror image G20 into a planar image to generate the upper unfolded image G31.

Then, the image conversion unit 123 generates the recognition image G30 by combining the right door unfolded image G32, the front unfolded image G33, the left door unfolded image G34, and the top unfolded image G31 with the distortion removed.

The image conversion unit 123 converts an image of food that is placed on the top surface of the top shelf 22U, which is difficult to photograph directly from the lens 52, into a flat image from the mirror image G20 to create an upper-level expanded image G31, which can be combined with other flat images to generate a recognition image G30 that shows the entire food or item.

Figure 7 is a flowchart showing an example of the in-fridge management process performed by the control unit 10. This process starts when the door 21 is opened.

First, the image capture unit 121 acquires the opening degree (angle) of the door 21 detected by the opening degree sensor 40 (S11). The image capture unit 121 calculates the angular velocity of the door 21 from the rate of change of the detection value (angle) of the opening degree sensor 40 (S12). The rate of change of the angle can be the rate of change from the previous value or the rate of change within a predetermined time (e.g., 100 ms). The angular velocity is a positive value in the door-opening direction and a negative value in the door-closing direction.

The image capture unit 121 determines whether the angular velocity of the door 21 is in the door-opening direction (positive value) (S13). If it is in the door-opening direction (S13: YES), the process proceeds to step S14. If it is in the door-closing direction (S13: NO), the process ends.

In step S14, it is determined whether the opening angle of the door 21 is equal to or greater than a preset threshold value Th1. If the opening angle of the door 21 is equal to or greater than the threshold value Th1 (S14: YES), the process proceeds to step S15. If the opening angle of the door 21 is less than the threshold value Th1 (S14: NO), the process returns to step S11 and repeats the above process. The threshold value Th1 is set, for example, to a value (for example, a value of around 100 degrees) at which food placed in the pocket 23 of the door 21 can be photographed.

In step S15, it is determined whether the door 21 has stopped. If the door 21 has stopped (S15: YES), the process proceeds to step S16. If the door 21 is opening or closing (S15: NO), the process returns to step S11 and the above process is repeated.

In step S16, the image capturing unit 121 captures the visible light image captured by the camera 50 and stores the captured visible light image in the image buffer 122 as a camera image GC.

The position where the door 21 stops is an opening angle equal to or greater than the threshold value Th1, which allows the pockets 23 inside the door 21 to be photographed, so that more food inside the fridge can be photographed. In addition, the photographing unit 121 photographs with the camera 50 after the movement of the door 21 has stopped, so blurring of the image of the food stored in the pockets 23, etc. is reduced, and clear still images can be obtained.

The image conversion unit 123 reads the camera image GC taken in step S16 from the image buffer 122 and converts the fisheye image into a planar image (S17). As shown in FIG. 6, the image conversion unit 123 extracts the mirror image G20, the right door image G11, the interior front image G12, and the left door image G13 from the fisheye camera image GC, converts each of these images G11 to G13 into a planar image, and generates the recognition image G30 by combining these converted planar images with the upper expanded image G31.

When the learning-based image recognition unit 1241 receives the recognition image G30 generated by the image conversion unit 123, it causes the machine learning model to perform image recognition of the food, and outputs the recognition result in text (S18). When the rule-based image recognition unit 1242 receives the recognition image G30 generated by the image conversion unit 123, it performs rule-based image recognition of the food, and outputs the recognition result in text (S19).

The processing order of the learning-based image recognition unit 1241 and the rule-based image recognition unit 1242 is not limited to the above, and they may be executed in parallel.

The recognized food storage locations are obtained from the table control unit 125 (S20). FIG. 8 shows an example of a storage location management table T1 in its initial state. The storage location management table T1 sets a priority order for the location in the refrigerator for each food. Table T1 is created in advance by determining the location that is easiest to recognize for each food through experiments. For example, the manufacturer of refrigerator 1 may output the recognition accuracy while changing the storage location for each food item to find out which location for each food item will result in the highest recognition accuracy. Priority 1 is the highest, followed by priority 2, priority 3, and so on.

The guidance unit 126 notifies the user of the appropriate storage location for each food item by light and sound using the notification means 30 (S21). The guidance unit 126 references the storage locations of the foods recognized by the recognition unit 124 from the table control unit 125 and notifies the user by light and sound. For example, when the storage location management table T1 is in the initial state shown in FIG. 8, the guidance unit 126 notifies the user by activating the speaker 31 and turning on the light 32 of the guided storage locations (22M, 22U, 23L) so that the natto 119 is placed on the middle shelf 22M of the refrigerator, the beer 120 is placed on the upper shelf 22U of the refrigerator, and the milk 121 is placed on the lower shelf 23L of the door pocket.

When the recognition result output unit 127 receives the recognition result from the learning-based image recognition unit 1241 and the recognition result from the rule-based image recognition unit 1242, it calculates the type and quantity of food from the text of the recognition result and generates a food list 72 (S22). The recognition result output unit 127 excludes items other than food from the recognition result from the food list 72.

The recognition result output unit 127 can determine the type and number of foods by performing a logical AND between the recognition result of the learning-based image recognition unit 1241 and the recognition result of the rule-based image recognition unit 1242. Alternatively, the recognition result output unit 127 may correct the recognition result of the learning-based image recognition unit 1241 with the recognition result of the rule-based image recognition unit 1242. The recognition result output unit 127 transmits the generated food list 72 to a pre-configured mobile terminal 7.

Figure 9 shows an example of a food list 72 displayed on the mobile terminal 7. The mobile terminal 7 is, for example, a smartphone. The mobile terminal 7 is not limited to a smartphone, and may be a tablet terminal, a wearable terminal, a laptop terminal, or the like. An application (not shown) for managing the inside of the refrigerator that runs on the mobile terminal 7 communicates with the refrigerator 1 regularly or irregularly via the network CN.

The received food list 72 is displayed on the display screen 71 of the mobile device 7. The food list 72 is a text that pairs food types with quantities. By referring to the food list 72, the user of the refrigerator 1 can quickly and easily understand the types and quantities of food stored in the refrigerator 1.

Although an example has been shown in which the photographing unit 121 photographs the interior of the refrigerator 1, and then performs image recognition of foods and generation and transmission of the food list 72, this is not limiting. For example, the control unit 10 may store the food list 72 calculated from the recognition results of the camera image GC in the storage device 12, and transmit the latest food list 72 to the mobile terminal 7 when a request for the food list 72 is received from the mobile terminal 7. This allows the user of the refrigerator 1 to quickly know how many and what types of foods are inside the refrigerator even when they are out, by referring to the mobile terminal 7.

As described above, in this embodiment, the priority of the placement location is defined for each food item so that the recognition accuracy is increased, and stored in a table. When the user puts food in the food item, this table is referenced and the user is notified of the recommended placement location for each food item, thereby improving the accuracy of identifying the food items.

In this embodiment, the machine learning model of the recognition unit 124 can be updated to the latest machine learning model in response to changes or additions to food packaging, and for example, a machine learning model received from a server (not shown) may be updated as the machine learning model of the learning-based image recognition unit 1241.

The table control unit 125 determines in advance the placement location for each food item that will provide the highest recognition accuracy through experiments, etc., but the user may set the placement location priority for each food item using a mobile terminal 7, etc.

The guidance unit 126 notifies the user of the location of food by sound and light, but the notification may be changed to only sound or light depending on the user's preference.

Using Figures 10 to 12, we will explain an application example in which food is ordered from an online supermarket based on a food list 72.

In this application example, the user of refrigerator 1 presets the types and quantities of food items that the user wishes to keep in refrigerator 1 at all times in stock table T2 (see FIG. 12) and stores the preset in storage device 12. The control unit 10 of refrigerator 1 periodically or irregularly compares the latest food list 72 with stock table T2, extracts foods that are in short supply from the foods registered in stock table T2, and places an order for them from the online supermarket.

Figure 12 is a diagram showing an example of the stock table T2. The stock table T2 is a table that is set in advance by the user of the refrigerator 1 via the mobile terminal 7 or the like. The stock table T2 includes, in one record, an item C21 that stores the type of food, a constant C22 that stores the quantity of the food that should be stored at all times, and a supplier C23 that stores the site from which the food is ordered.

Figure 10 is a flowchart showing an example of food ordering processing performed by the control unit 10. The explanation is based on the assumption that the user is about to open the door 21 of the refrigerator 1 and place food inside. Therefore, the explanation of the steps for determining the state of the door 21 will be omitted here.

The control unit 10 acquires a visible light image from the camera 50 and stores the acquired visible light image in the image buffer 122 as a camera image GC (S31).

The image conversion unit 123 reads the camera image GC captured in step S16 from the image buffer 122 and converts the fisheye image into a planar image (S32).

When the recognition image G30 is input, the learning-based image recognition unit 1241 causes the machine learning model to perform image recognition of the food and outputs the recognition result in text (S33). When the recognition image G30 is input, the rule-based image recognition unit 1242 performs rule-based image recognition of the food and outputs the recognition result in text (S34). The control unit 10 obtains the location of the recognized food from the table control unit 125 (S35).

The control unit 10 notifies the user of the placement location (S36). The control unit 10 generates a food list 72 (S37). The control unit 10 causes the recognition result output unit 127 to read the stock table T2 and compare the quantity of food listed in the food list 72 with the constant C22 of the stock table T2 (S38).

The recognition result output unit 127 determines whether there is any food whose quantity in the food list 72 is less than the value set in the constant C22 in the stock table T2 (S39). If the control unit 10 finds a food whose quantity (current stock) is less than the constant C22 in the stock table T2 (S39: YES), it proceeds to step S40, otherwise it proceeds to step S41.

In step S40, the recognition result output unit 127 calculates the difference between the current quantity of the food item in short supply and the constant C22 as the order quantity, generates an order list 73 (see FIG. 11), and transmits it to a pre-configured mobile terminal 7.

The user of refrigerator 1 checks the order list 73 displayed on the mobile terminal 7 and places an online order for food from a pre-set supplier C23. However, in step S41, since no missing food items are found, the food list 72 is sent to the pre-set mobile terminal 7.

The mobile terminal 7 to which the food list 72 is sent may be different from the mobile terminal 7 to which the order list 73 is sent. For example, if the user of the refrigerator 1 and the manager with food purchasing authority are different people, the food list 72 may be sent to the user of the refrigerator 1, and the order list 73 (or the order list 73 and the food list 72) may be sent to the manager with food purchasing authority. This allows, for example, a parent to remotely manage the inventory in a refrigerator used by a child who lives far away, and monitor the child's diet and nutritional status. Alternatively, a child can remotely manage the inventory in a refrigerator used by a parent who lives far away, and estimate and monitor the parent's dietary status. Furthermore, a coach can remotely manage the inventory in a refrigerator used by a user who is on a diet.

Figure 11 shows an example of an order list 73 displayed on the mobile device 7. The order list 73 received from the refrigerator 1 is displayed on the display screen 71 of the mobile device 7. The order list 73 displays pairs of items C21 that have not reached the constant C22 in the stock table T2 and the quantity that is insufficient for the constant C22.

Below the order list 73, a supplier 74 and an order button 75 are displayed. When an operation of the order button 75 by the user of the refrigerator 1 is detected, the contents of the order list 73 are sent to the supplier 74.

Note that while FIG. 11 shows an example in which a single supplier 74 is displayed, multiple suppliers 74 may be displayed so that the user can select one when placing an order. Alternatively, a supplier may be selected from multiple suppliers registered in advance according to the type of food.

The timing when the recognition result output unit 127 generates the order list 73 is not limited to immediately after the camera 50 captures an image of the warehouse interior, but can be performed periodically or irregularly.

The second embodiment will be described with reference to Figures 13 and 14. In the following embodiments, including this embodiment, the differences from the first embodiment will be mainly described. In this embodiment, when a user places food in a location different from the location recommended by the refrigerator 1, the storage location management table T1 is modified to meet the user's wishes, thereby further improving usability for the user.

In other words, in this embodiment, if the user places food in a location other than the notified location with priority 1, the priority of the location where the food was placed is changed, and table T1 is updated to be more suitable for the user.

Figure 13 is a flowchart showing an example of the refrigerator management process. This process will also be described assuming that the user opens the door 21 of the refrigerator 1 and attempts to store food inside.

The control unit 10 acquires a visible light image from the camera 50 and stores the acquired visible light image in the image buffer 122 as a camera image GC (S51). The image conversion unit 123 converts the camera image GC into a planar image (S52).

When the recognition image G30 is input, the learning-based image recognition unit 1241 causes the machine learning model to perform image recognition of the food and outputs the recognition result in text (S53). When the recognition image G30 is input, the rule-based image recognition unit 1242 performs rule-based image recognition of the food and outputs the recognition result in text (S54). The control unit 10 obtains the location of the recognized food from the table control unit 125 (S55). The control unit 10 notifies the user of the location (S56).

The control unit 10 detects the location where the user actually placed the food by processing the image captured by the camera 50 (S57), and determines whether the actual location matches the location indicated by the refrigerator 1 (the location notified by the notification means 30, also called the recommended location) (S58).

If the location where the user actually placed the food matches the location indicated by the refrigerator 1 (S58: YES), the control unit 10 generates a food list 72 (S59). The control unit 10 outputs the food list 72 (S60).

In contrast, if the location where the user actually placed the food differs from the location suggested by the refrigerator 1 (S58: NO), the detection parameters (e.g., thresholds) included in the location detection process (S57) are updated (S61). The detection parameters are parameters of the location detection unit 1243, and are determined for each location. By increasing the detection parameters, the location management table T1 is updated (S62), and a location that matches the user's usage can be suggested.

The following describes a case where a user places milk inside the refrigerator. Referring to the storage location management table T1 in FIG. 8, the storage location with the highest priority for milk is the lower door pocket 23L. However, if the user places milk in the upper refrigerator compartment 22U, which has a lower priority than the lower door pocket 23L, the storage location detection process S57 sets the detection parameter for the upper refrigerator compartment 22U to the highest value. As a result, the upper refrigerator compartment 22U becomes the highest priority location for milk, and the table is updated from the next time onwards (FIG. 14).

This embodiment configured in this way also achieves the same effects as the first embodiment. Furthermore, in this embodiment, if the location where the user places the food inside the refrigerator differs from the location recommended by the refrigerator 1, the priority of the locations is changed and table T1 is updated, and the detection parameters are adjusted to improve the accuracy of recognition of the location where the food was actually placed. This makes it possible to introduce a location that will improve the accuracy of food recognition according to the user's preferences and behavior.

The third embodiment will be described with reference to FIG. 15. In this embodiment, a method for handling a case where the storage location recommended to the user (notified storage location) is already filled with other food and new food cannot be stored will be described.

In this embodiment, the table control unit 125 references the food storage location recognized by the recognition unit 124, but if there is no space to place the food in the referenced location, the storage location with the next highest priority is selected. The presence or absence of space to place the food is determined by the storage location detection unit 1243.

Figure 15 is a flowchart showing an example of the interior management process performed by the control unit 10. In the following, the door 21 of the refrigerator 1 is open, and the process in which the user is about to store food is also illustrated in the flowchart.

Steps S11 to S20 and S21 to S23 of this process are similar to steps S11 to S20 and S21 to S23 described in FIG. 7, so their explanation will be omitted. In this process, new steps S71 and S72 have been added between steps S20 and S21.

In step S71, the food storage location recognized by the recognition unit 124 is referenced from the table control unit 125, and if there is space to place the food in the referenced location, the process proceeds to step S21, and if there is no space, the process proceeds to step S72. In step S72, the next priority for the food storage location is referenced from table T1.

This embodiment configured in this way also achieves the same effect as the first embodiment. Furthermore, in this embodiment, if there is no space to place food in the location notified to the user, the table control unit 125 can refer to the next priority for the food placement location and notify the user of a suitable placement location for that food, thereby improving usability for the user.

The fourth embodiment will be described with reference to Figures 16 and 17. In this embodiment, the process of adding new food to the storage location management table T1 will be described when storing food in the refrigerator 1 that is not registered in the storage location management table T1.

Figure 16 is a flowchart showing an example of the warehouse management process performed by the control unit 10. Steps S11 to S23 of this process are similar to steps S11 to S23 described in Figure 7, so their explanation is omitted. In this embodiment, new steps S81 and S82 are added between steps S19 and S20.

In step S81, it is determined whether the food recognized by the recognition unit 124 has already been registered in table T1 managed by the table control unit 125. If the food has already been registered in table T1, the process proceeds to step S20. If the food has not yet been registered, the process proceeds to step S82.

In step S82, foods that are not registered in table T1 are registered with the same priority as the most similar food among the foods registered in table T1. The similarity of foods is determined by the recognition unit 124.

Figure 17 shows a modified example of this embodiment. In the modified warehouse management process, steps S91 and S92 are provided instead of steps S81 and S82 in Figure 16. However, steps S91 and S81 are substantially the same.

In step S92, foods that are not registered in table T1 of the table control unit 125 are registered in table T1 by the user manually setting the priority order from the mobile terminal 7.

This embodiment configured in this way also achieves the same effects as the first embodiment. Furthermore, in this embodiment, when a food item not registered in table T1 is to be stored in the refrigerator, the user can set the priority of the unregistered food item in table T1 from the mobile terminal 7, improving usability for the user.

The fifth embodiment will be described with reference to FIG. 18. This embodiment shows the fifth embodiment of the present invention, and illustrates an example in which the user notifies the control unit 10 of the correct answer when a food item is incorrectly recognized. In this embodiment, when the food item recognition result is incorrect, the user instructs the control unit 10 of the correct answer, thereby improving the recognition accuracy from the next time onwards.

Figure 18 is a flowchart showing an example of the warehouse management process performed by the control unit 10. Steps S11 to S23 of this process are similar to steps S11 to S23 described in Figure 7, so a description thereof will be omitted. In this process, new steps S101 to S103 have been added between steps S19 and S20.

The control unit 10 of the refrigerator 1 notifies the user by voice of the food name recognized by the recognition unit 124 (S101). When the user inputs whether the read-out food name is correct or not, the control unit 10 determines whether the recognized food name is correct (S102). If the recognized food name is correct, the process proceeds to step S20, and if it is not correct, the process proceeds to step S103.

Methods by which the user can instruct the control unit 10 on the correct food name include, for example, the user manually inputting the correct food name from the mobile terminal 7, and the user verbally stating the correct food name.

When the correct answer for the incorrectly recognized food is input by the user to the control unit 10 (S103), the machine learning model of the learning-based image recognition unit 1241 that recognizes the food is updated based on the correct answer input by the user (S104).

This embodiment configured in this way also achieves the same effect as the first embodiment. Furthermore, in this embodiment, if the food recognized by the recognition unit 124 is incorrect, the user can announce the correct food name, which improves the recognition accuracy of the incorrectly recognized food from the next time onwards.

This concludes the explanation of the present invention, but the present invention is not limited to the above-mentioned embodiments and includes various modifications. For example, the above-mentioned embodiments have been described in detail to provide a better understanding of the present invention, and are not necessarily limited to those having all of the configurations described above.

It is also possible to replace part of the configuration of one embodiment with the configuration of another embodiment. It is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to delete part of the configuration of each embodiment, add other configurations, or replace it with other configurations.

The above configurations, functions, processing units, processing means, etc. may be realized in hardware, in part or in whole, for example by designing them as integrated circuits. Also, the above configurations, functions, etc. may be realized in software, by a processor interpreting and executing a program that realizes each function. Information on the programs, tables, files, etc. that realize each function can be stored in storage devices such as non-volatile semiconductor memory, hard disk drives, and SSDs (Solid State Drives), or in computer-readable non-transitory data storage media such as IC cards, SD cards, and DVDs.

In addition, the control lines and information lines shown are those considered necessary for the explanation, and not all control lines and information lines on the product are necessarily shown. In reality, it can be assumed that almost all components are interconnected.

Furthermore, the above-mentioned embodiments can be combined as appropriate, and such combinations of embodiments are also included within the scope of the present invention.

1: refrigerator, 7: mobile terminal, 8: web server, 9: computer, 10: control unit, 12: storage device, 20: refrigerator body, 21: door, 22: shelf, 23: door pocket, 30: notification means, 40: opening sensor, 50: camera, 60: mirror, photography unit 121, 123: image conversion unit, 124: recognition unit, 125: table control unit, 126: guidance unit

Claims (7)

The storage body,
A camera unit that photographs the inside of the storage body;
A notification unit that notifies a user of a specific storage unit among the multiple storage units of the storage body;
a control unit that, when a target item is stored in the storage main body, photographs the target item with the camera unit, recognizes the target item through image recognition processing, selects the predetermined storage unit for storing the recognized target item based on a predetermined table that manages the priority order of storage units for storing items, and notifies the notification unit of the selected predetermined storage unit;
Equipped with
In the predetermined table, the priority order of the storage units is set in order of increasing recognition accuracy of the object,
When the target item is stored in a storage section other than the predetermined storage section, the control section updates the priority of the other storage section for the target item in the predetermined table so as to increase the priority, and further updates parameters of the image recognition process so as to improve recognition accuracy of the target item stored in the other storage section.
Storage facility . The storage body,
A camera unit that photographs the inside of the storage body;
A notification unit that notifies a user of a specific storage unit among the multiple storage units of the storage body;
a control unit that, when a target item is stored in the storage main body, photographs the target item with the camera unit, recognizes the target item through image recognition processing, selects the predetermined storage unit for storing the recognized target item based on a predetermined table that manages the priority order of storage units for storing items, and notifies the notification unit of the selected predetermined storage unit;
Equipped with
In the predetermined table, the priority order of the storage units is set in order of increasing recognition accuracy of the object,
when the target item is stored in a storage section other than the specified storage section, the control section updates the priority of the other storage section for the target item in the specified table to be higher, and further updates parameters of the image recognition process to improve recognition accuracy of the target item stored in the other storage section;
When the control unit determines that there is no space for storing the target item in the predetermined storage unit, the control unit selects , from the predetermined table, a storage unit having a lower priority than the priority of the predetermined storage unit as a new predetermined storage unit.
Storage facility . The storage body,
A camera unit that photographs the inside of the storage body;
A notification unit that notifies a user of a specific storage unit among the multiple storage units of the storage body;
a control unit that, when a target item is stored in the storage main body, photographs the target item with the camera unit, recognizes the target item through image recognition processing, selects the predetermined storage unit for storing the recognized target item based on a predetermined table that manages the priority order of storage units for storing items, and notifies the notification unit of the selected predetermined storage unit;
Equipped with
In the predetermined table, the priority order of the storage units is set in order of increasing recognition accuracy of the object,
when the target item is stored in a storage section other than the specified storage section, the control section updates the priority of the other storage section for the target item in the specified table to be higher, and further updates parameters of the image recognition process to improve recognition accuracy of the target item stored in the other storage section;
When the target item is an unregistered item that is not registered in the predetermined table, the control unit selects the predetermined storage unit according to a priority order of an item that is most similar to the target item among the items registered in the predetermined table.
Storage facility . The storage body,
A camera unit that photographs the inside of the storage body;
A notification unit that notifies a user of a specific storage unit among the multiple storage units of the storage body;
a control unit that, when a target item is stored in the storage main body, photographs the target item with the camera unit, recognizes the target item through image recognition processing, selects the predetermined storage unit for storing the recognized target item based on a predetermined table that manages the priority order of storage units for storing items, and notifies the notification unit of the selected predetermined storage unit;
Equipped with
In the predetermined table, the priority order of the storage units is set in order of increasing recognition accuracy of the object,
when the target item is stored in a storage section other than the specified storage section, the control section updates the priority of the other storage section for the target item in the specified table to be higher, and further updates parameters of the image recognition process to improve recognition accuracy of the target item stored in the other storage section;
When the target item is an unregistered item that is not registered in the predetermined table, the control unit selects the predetermined storage unit according to a priority order designated by a user.
Storage facility . The storage body,
A camera unit that photographs the inside of the storage body;
A notification unit that notifies a user of a specific storage unit among the multiple storage units of the storage body;
a control unit that, when a target item is stored in the storage main body, photographs the target item with the camera unit, recognizes the target item through image recognition processing, selects the predetermined storage unit for storing the recognized target item based on a predetermined table that manages the priority order of storage units for storing items, and notifies the notification unit of the selected predetermined storage unit;
Equipped with
In the predetermined table, the priority order of the storage units is set in order of increasing recognition accuracy of the object,
when the target item is stored in a storage section other than the specified storage section, the control section updates the priority of the other storage section for the target item in the specified table to be higher, and further updates parameters of the image recognition process to improve recognition accuracy of the target item stored in the other storage section;
When the target item is an unregistered item that is not registered in the predetermined table, the control unit selects the predetermined storage unit in accordance with a priority order designated by a user,
The control unit accepts corrections from a user to a recognition result obtained by the image recognition processing , and updates a machine learning model used in the image recognition processing.
Storage facility . The storage body,
A camera unit that photographs the inside of the storage body;
A notification unit that notifies a user of a specific storage unit among the multiple storage units of the storage body;
a control unit that, when a target item is stored in the storage main body, photographs the target item with the camera unit, recognizes the target item through image recognition processing, selects the predetermined storage unit for storing the recognized target item based on a predetermined table that manages the priority order of storage units for storing items, and notifies the notification unit of the selected predetermined storage unit;
Equipped with
In the predetermined table, the priority order of the storage units is set in order of increasing recognition accuracy of the object,
when the target item is stored in a storage section other than the specified storage section, the control section updates the priority of the other storage section for the target item in the specified table to be higher, and further updates parameters of the image recognition process to improve recognition accuracy of the target item stored in the other storage section;
The notification units are individually disposed in the plurality of storage units,
A notification unit disposed in the predetermined storage unit is activated to notify the user.
Storage facility . The storage body,
A camera unit that photographs the inside of the storage body;
A notification unit that notifies a user of a specific storage unit among the multiple storage units of the storage body;
a control unit that, when a target item is stored in the storage main body, photographs the target item with the camera unit, recognizes the target item through image recognition processing, selects the predetermined storage unit for storing the recognized target item based on a predetermined table that manages the priority order of storage units for storing items, and notifies the notification unit of the selected predetermined storage unit;
Equipped with
In the predetermined table, the priority order of the storage units is set in order of increasing recognition accuracy of the object,
when the target item is stored in a storage section other than the specified storage section, the control section updates the priority of the other storage section for the target item in the specified table to be higher, and further updates parameters of the image recognition process to improve recognition accuracy of the target item stored in the other storage section;
The notification units are individually disposed in the plurality of storage units,
A notification unit disposed in the predetermined storage unit is activated to notify the user,
The notification unit includes light emitters arranged in a matrix on the floor surface of the storage unit.
Storage facility .

Images