JP7584035B2 - refrigerator - Google Patents

Description

The present invention relates to a refrigerator.

Conventionally, a refrigerator has been proposed in which an interior camera is installed in the center of a food storage compartment, the interior camera captures images of the interior of the storage compartment when the door of the storage compartment is opened and closed, and an image of when the food was stored and a current image are output side by side from the captured images in chronological order (see, for example, Patent Document 1). In the refrigerator described in Patent Document 1, depending on the arrangement of the food in the storage compartment, the food may overlap in the direction of the image capture by the interior camera, making it impossible to recognize some of the food from the captured images.
In response to this, a refrigerator has been proposed that is equipped with a switch that detects the opening and closing of the storage compartment door and a camera whose imaging range is the opening of the storage compartment, so that when the door is opened, an image is captured of food being stored or removed through the opening, rather than of food stored in the storage compartment (see, for example, Patent Document 2).

JP 2016-57022 A JP 2015-81762 A

The present invention aims to provide a refrigerator that can more reliably obtain information about food stored in the storage compartment.

A refrigerator according to a first aspect of the present invention includes a storage compartment having an opening, a door provided in the storage compartment, an opening/closing sensor for detecting an open/closed state of the door, a camera for capturing an image of an image capturing range including the opening, and a memory unit, and the camera captures images a plurality of times from when the opening/closing sensor detects that the door has been switched from a closed state to an open state until the opening/closing sensor detects that the door has been switched from an open state to a closed state, and captures images from a first image capturing time point when a first predetermined time has elapsed from when the opening/closing sensor detects that the door has been switched from a closed state to an open state , the first image capturing time point being set assuming a time when food to be stored in the storage compartment will pass through the opening, and the camera captures images of a previous image capturing time point when the opening/closing sensor detects that the door has been switched from an open state to a closed state. The food information acquisition unit is configured to repeatedly capture images using the camera until the opening/closing sensor detects that the door has switched from an open state to a closed state, and store a plurality of captured images in the memory unit as candidate images at the time of leaving the warehouse together with the capture time, and when the opening/closing sensor detects that the door has switched from an open state to a closed state , store in the memory unit as a candidate image at the time of leaving the warehouse, the candidate image at the time of leaving the warehouse, which was captured at a second capturing time point which is a second predetermined time before the time when the opening/closing sensor detects that the door has switched from an open state to a closed state, as a candidate image at the time of leaving the warehouse.
A refrigerator according to a second aspect of the present invention includes a storage chamber having an opening, a door provided in the storage chamber, an opening/closing sensor for detecting an open/closed state of the door, a camera for capturing an image of an image capturing range including the opening, and a memory unit, and the camera captures images a plurality of times from when the opening/closing sensor detects that the door has switched from a closed state to an open state until the opening/closing sensor detects that the door has switched from an open state to a closed state, and the camera captures images a plurality of times from when the opening/closing sensor detects that the door has switched from an open state to a closed state, and the camera captures images a plurality of times from when the opening/closing sensor detects that the door has switched from a closed state to an open state until the opening/closing sensor detects that the door has switched from a closed state to an open ... food to be stored in the storage chamber passes through the opening. a food information acquisition unit that stores in the memory unit an image taken at a first imaging time point when a first predetermined time has elapsed as an entry image, and stores in the memory unit an image taken at a second imaging time point which is a second predetermined time before the time point when the opening/closing sensor detects that the door has switched from the open state to the closed state and which is after the first imaging time point and which is set assuming the time when food being removed from the storage chamber passes through the opening , and acquires information about the food being entered into the storage chamber and the food being removed from the storage chamber based on the entry image and the exit image.

According to the present invention, it is possible to obtain information on food items stored in the storage room more reliably.

FIG. 1 is an explanatory diagram of a configuration for checking the status of food stored in a refrigerator installed in a house using a terminal device. Diagram of the refrigerator's appearance An explanatory diagram showing the internal structure of a refrigerator in a cross-sectional view from the right side. An explanatory diagram showing the internal structure of the refrigerator compartment as a cross-sectional view from the front. Control unit configuration diagram 1 is a flowchart showing the overall processing by the control unit. Flowchart of imaging process Flowchart of weight information processing Flowchart of analysis process of images taken at the time of storage Flowchart of analysis process of images at the time of leaving the warehouse Flowchart of weight sensor correction process First flow chart of food management process in warehouse Second flow chart of food management process in warehouse List of food in the warehouse and list of expiration dates 11 is a flowchart of another embodiment of the imaging process.

The first invention comprises a storage chamber having an opening, an opening/closing unit which opens and closes the opening, an opening/closing sensor which detects the open/closed state of the opening/closing unit, a camera which captures an image of an imaging range including the opening, a memory unit, and a food information acquisition unit which captures an image by the camera at a first imaging time point which is set to assume a time point when food to be stored in the storage chamber will pass through the opening between when the opening/closing sensor detects that the opening/closing unit has switched from a closed state to an open state and when the opening/closing sensor detects that the opening/closing unit has switched from an open state to a closed state, and stores the image captured at the first imaging time point in the memory unit as an entry image, and which captures an image by the camera at a second imaging time point which is set to assume a time point when food to be removed from the storage chamber will pass through the opening, and stores the image captured at the second imaging time point in the memory unit as an exit image, and acquires information about the food to be stored in the storage chamber and the food to be removed from the storage chamber based on the entry image and the exit image.
According to the first aspect of the present invention, the food information acquisition unit stores in the storage unit an entry image captured at a first imaging time point when it is assumed that the food being entered into the storage unit passes through the opening of the storage unit, and an exit image captured at a second imaging time point when it is assumed that the food being exited from the storage unit passes through the opening. The food information acquisition unit can more reliably acquire information on the food being entered into the storage unit and the food being exited from the storage unit based on the entry image and the exit image that are likely to include image portions of the food.

A second invention includes an illuminance sensor that detects the illuminance within the storage chamber, and an imaging time point setting unit that sets a first reference illuminance and a second reference illuminance based on the detected illuminance of the illuminance sensor at the time point when the opening/closing sensor detects that the opening/closing section has switched from a closed state to an open state, sets the first imaging time point based on the time point when the detected illuminance of the illuminance sensor becomes equal to or lower than the first reference illuminance, and sets the second imaging time point based on the time point when the detected illuminance of the illuminance sensor becomes equal to or higher than the second reference illuminance after becoming equal to or lower than the first reference illuminance.
According to the second invention, the illuminance in the storage chamber changes depending on the presence of an object (food, a user's hand, etc.) in the storage chamber. Therefore, based on the change in illuminance in the storage chamber detected by the illuminance sensor, it is possible to appropriately set a first imaging time point that assumes the time when the food being stored in the storage chamber passes through the opening, and a second imaging time point that assumes the time when the food being removed from the storage chamber passes through the opening.

The third invention includes an imaging time setting unit that sets the first imaging time as a time when a first predetermined time has elapsed since the opening/closing sensor detected that the opening/closing section has switched from a closed state to an open state, and sets the second imaging time as a time that is a second predetermined time before the opening/closing sensor detected that the opening/closing section has switched from an open state to a closed state.
According to the third aspect of the present invention, the time from when the user opens the opening/closing part of the storage compartment to when the food is stored and the time from when the user removes the food from the storage compartment to when the user closes the opening/closing part can be expected to be relatively short and constant in order to maintain the cold state within the storage compartment. Therefore, the first image capture time point and the second image capture time point can be set by a simple configuration of setting the first and second predetermined times.

In a fourth invention, the food information acquisition unit repeatedly captures images using the camera and stores the captured images in the memory unit from the time when the first imaging time has passed until the opening/closing sensor detects that the opening/closing section has switched from an open state to a closed state, and when the opening/closing sensor detects that the opening/closing section has switched from an open state to a closed state, saves the captured image captured at a time point the second predetermined time before the time when the opening/closing sensor detected that the opening/closing section had switched from an open state to a closed state and stored in the memory unit as the exit image.
According to the fourth aspect of the present invention, the food information acquisition unit repeatedly captures images using the camera and stores the captured images in the storage unit from the time when the first image capturing time has passed until the opening/closing sensor detects that the opening/closing part has switched from the open state to the closed state. This allows the captured image captured at a time point a second predetermined time before the time when the opening/closing sensor detects that the opening/closing part has switched from the open state to the closed state to be stored as a warehouse-time image.

The following describes an embodiment of the present invention with reference to the drawings. Note that the present invention is not limited to this embodiment.

[1. Usage of refrigerator]
1 shows a usage state of a refrigerator according to this embodiment. Refrigerator 1 according to this embodiment is installed in house H, and has a function of communicating with a management server 520 via a gateway 500 and a communication network 510. Refrigerator 1 also has a function of communicating with a terminal device (smartphone, tablet terminal, etc.) 400. Terminal device 400 has a function of communicating with the management server 520 via the gateway 500 and the communication network 510.

The control unit 100 provided in the refrigerator 1 controls the overall operation of the refrigerator 1, and generates an internal food list 131 showing information about the food stored in the storage compartment of the refrigerator 1 and sends it to the management server 520. The refrigerator management application executed on the terminal device 400 acquires the internal food list 131 data from the management server 520, or acquires the internal food list 131 data from the refrigerator 1, and displays information about the food stored in the refrigerator 1 based on the internal food list 131 on the display unit. A user P of the refrigerator 1 can check the status of the food stored in the refrigerator 1 (food name, expiration date, remaining amount, etc.) by visually checking the display on the terminal device 400.

[2. Configuration of the refrigerator]
The configuration of the refrigerator 1 will be described with reference to Figures 2 to 5. Figure 2 is an explanatory diagram showing the external appearance of the refrigerator 1. As shown in Figure 2, the refrigerator 1 has a main box body 2 with an opening on the front side, and inside the main box body 2, a refrigeration chamber 10 (corresponding to a storage chamber of the present invention), an ice-making chamber 30, a switching chamber 20 that is adjacent to the ice-making chamber 30 and allows the temperature inside to be changed, a freezer chamber 40, and a vegetable chamber 50 are formed.

The opening at the front of the refrigerator compartment 10 is provided with a rotating right door 11 and a left door 12 (corresponding to the opening and closing part of the present invention). In addition, the switching compartment 20, the ice making compartment 30, the freezer compartment 40, and the vegetable compartment 50 are each provided with drawers 21, 31, 41, and 51 for storing food.

Next, FIG. 3 is an explanatory diagram showing the internal configuration of the refrigerator 1 in a cross-sectional view seen from the right side, and FIG. 4 is an explanatory diagram showing the internal configuration of the refrigerator compartment 10 in a cross-sectional view seen from the front. With reference to FIG. 3 and FIG. 4, the refrigerator compartment 10 is provided with an upper shelf 13, a middle shelf 14, and a lower shelf 15 that divide the refrigerator compartment 10 into sub-areas. The refrigerator compartment 10 is also provided with weight sensors 13a and 13b that detect the weight of food placed on the upper shelf 13, weight sensors 14a and 14b that detect the weight of food placed on the middle shelf 14, and weight sensors 15a and 15b that detect the weight of food placed on the lower shelf 15. The refrigerator compartment 10 is also provided with lighting units 16a to 16f that illuminate the interior of the refrigerator compartment 10, illuminance sensors 17a to 17c that detect the illuminance within the refrigerator compartment 10, and a camera 60.

The camera 60 is positioned near the opening 10a at the top of the refrigerator compartment 10 so that the opening 10a is included in the imaging range, and images the area from the opening 10a to the front ends of the shelves 13, 14, and 15. In addition, a compressor 61, a cooling fan 62, a cooler 63, and a condenser 64, which are auxiliary machines that make up the refrigeration cycle, are provided. Furthermore, a refrigerator compartment duct 70 and cold air outlets 71-73 for circulating cold air are provided at the back of the refrigerator compartment 10.

The refrigerator compartment 10 is provided with an opening/closing sensor 18b that detects whether the right door 11 is open or closed, and an opening/closing sensor 18a that detects whether the left door 12 is open or closed. The opening/closing sensor 18b outputs a closed detection signal when the right door 11 is closed, and outputs an open detection signal when the right door 11 is open. Similarly, the opening/closing sensor 18a outputs a closed detection signal when the left door 12 is closed, and outputs an open detection signal when the left door 12 is open.

The switching chamber 20 is provided with an opening/closing sensor 22 that detects whether the opening 20a is open or closed. The opening/closing sensor 22 outputs a closed detection signal when the drawer 21 is housed in the switching chamber 20, and outputs an open detection signal when the drawer 21 is pulled out of the switching chamber 20.

Similarly, the ice making compartment 30 is provided with an opening/closing sensor 32 that detects whether the opening 30a is opened or closed by the drawer 31. The freezer compartment 40 is provided with an opening/closing sensor 42 that detects whether the opening 40a is opened or closed by the drawer 41, and the vegetable compartment 50 is provided with an opening/closing sensor 52 that detects whether the opening 50a is opened or closed by the drawer 51. The opening/closing sensors 32, 42, and 52 output a closed detection signal and an open detection signal, similar to the opening/closing sensor 22.

Next, FIG. 5 is a configuration diagram of the control unit 100. Referring to FIG. 5, the control unit 100 is an electronic circuit unit that is composed of a CPU (Central Processing Unit) 110, an image processing unit 120, a memory unit 130, a wireless communication unit 140, a date and time recognition unit 141, and an interface circuit (not shown), etc.

The CPU 110 executes the control program for the refrigerator 1 stored in the memory unit 130, thereby functioning as a passing object detection unit 111, a storage capacity recognition unit 112, an entry/exit determination unit 113, a food information acquisition unit 114, a food information management unit 115, and an image capture time setting unit 116.

The control unit 100 is connected to the opening and closing sensors 18a, 18b, 22, 32, 42, 52, the illuminance sensors 17a, 17b, 17c, the weight sensors 13a, 13b, 14a, 14b, 15a, 15b, the camera 60, the lighting units 16a, 16b, 16c, 16d, 16e, 16f, and the refrigeration cycle auxiliary units 61, 62, 63, 64.

The control unit 100 receives detection signals from the open/close sensors 18a, 18b, 22, 32, 42, 52, the illuminance sensors 17a, 17b, 17c, and the weight sensors 13a, 13b, 14a, 14b, 15a, 15b. In addition, the control signals output from the control unit 100 control the operation of the lighting units 16a, 16b, 16c, 16d, 16e, 16f, and the refrigeration cycle auxiliary units 61, 62, 63, 64.

Furthermore, the image capturing by the camera 60 is controlled via the image processing unit 120 by a control signal output from the control unit 100. The image processing unit 120 converts the video signal output from the camera 60 into a digital gradation signal to generate a captured image, and stores the captured image data in the image memory 121. The image processing unit 120 processes the captured image stored in the image memory 121 in response to a control signal output from the CPU 110.

The date and time recognition unit 141 recognizes the current date and time (date and time) by measuring a clock signal output from a clock circuit (not shown) provided in the control unit 100. Note that the current date and time may be recognized by acquiring information on the current date and time from the management server 520 through communication with the management server 520.

In addition to the data of the in-fridge food list 131, the memory unit 130 stores data of incoming food information 132, which records information on foods stored in the refrigerator compartment 10, and outgoing food information 133, which records information on foods that have been taken out of the refrigerator compartment 10. In addition, the memory unit 130 stores data of sample images 134 of various foods, and an expiration date list 135, which shows general expiration dates for various foods.

The passing object detection unit 111 detects objects passing through the opening 10a of the refrigerator compartment 10 based on changes in illuminance inside the refrigerator compartment 10 detected by the illuminance sensors 17a, 17b, and 17c. The storage amount recognition unit 112 recognizes the amount of food stored on the upper shelf 13, middle shelf 14, and lower shelf 15 of the refrigerator compartment 10 by weight based on detection signals from the weight sensors 13a, 13b, 14a, 14b, 15a, and 15b.

The storage amount recognition unit 112 recognizes the weight of food stored on the upper shelf 13 based on the detection signals of the weight sensors 13a and 13b, and recognizes the weight of food stored on the middle shelf 14 based on the detection signals of the weight sensors 14a and 14b. The storage amount recognition unit 112 also detects the weight of food stored on the lower shelf 15 based on the detection signals of the weight sensors 15a and 15b.

When the passing object detection unit 111 detects an object passing through the opening 10a of the refrigerator compartment 10, the entry/exit determination unit 113 determines whether food has been entered into or removed from the refrigerator compartment 10 based on the change in the amount (weight) of food placed on each shelf 13, 14, 15 recognized by the storage capacity recognition unit 112.

The food information acquisition unit 114 uses the camera 60 to capture an image of an object that passes through the opening 10a of the refrigerator compartment 10. Then, when the in/out determination unit 113 determines that food has been stored in or removed from the refrigerator compartment 10, it extracts an image portion of the food from the captured image.

The food information acquisition unit 114 acquires information such as the name, content, place of origin, and expiration date of food that has been stored or shipped by recognizing characters contained in the image portion of the food or by extracting features of the image portion of the food. The food information acquisition unit 114 then records the acquired information in stored food information 132 for food determined to be stored (stored food), and in shipped food information 133 for food determined to be shipped (shipped food).

The food information management unit 115 updates the food information recorded in the in-fridge food list 131 based on the weight of the food in the refrigerator compartment 10 recognized by the storage capacity recognition unit 112, the incoming food information 132 and the outgoing food information 133 recorded by the food information acquisition unit 114, etc.

The image capturing time setting unit 116 sets the time when the camera 60 captures an image between the time when the right door 11 or the left door 12 is switched from a closed state to an open state by the opening/closing sensors 18a, 18b and the time when it is detected that both the right door 11 and the left door 12 are closed. The image capturing time setting unit 116 sets a first image capturing time that assumes the time when food entering the refrigerator compartment 10 passes through the opening 10a, and a second image capturing time that assumes the time when food removed from the refrigerator compartment 10 passes through the opening 10a.

[3. Processing by the control unit]
Next, the process for managing food stored in the refrigerator compartment 10, which is executed by the control unit 100, will be described with reference to FIGS.

Figure 6 is a flow chart showing an overview of a series of processes executed by the control unit 100. The control unit 100 executes "imaging process" in step S1, "weight information process" in step S2, "image analysis process" in step S3, "weight sensor correction process" in step S4, and "in-storage food management process" in step S5. Each process is described in detail below.

[3-1. Imaging Processing]
The procedure for executing the "imaging process" will be described with reference to the flowchart shown in Fig. 7. The "imaging process" is executed by the passing object detection unit 111, the food information acquisition unit 114, and the imaging time point setting unit 116. The food information acquisition unit 114 captures images with the camera 60 at a first imaging time point and a second imaging time point during the period from when it recognizes in step S10 that one or both of the right door 11 and the left door 12 of the refrigerator compartment 10 have switched from a closed state to an open state based on the detection signals of the opening/closing sensors 18a, 18b until it recognizes in step S15 that both the right door 11 and the left door 12 have become closed.

When the food information acquisition unit 114 recognizes in step S10 that one or both of the right door 11 and the left door 12 of the refrigerator compartment 10 have switched from a closed state to an open state, the processing proceeds to step S10. Steps S11 to S14 are processes performed by the passing object detection unit 111 and the image capture time setting unit 116. In this embodiment, the image capture time setting unit of the present invention is configured by a combination of the passing object detection unit 111 and the image capture time setting unit 116.

In step S11, the passing object detection unit 111 detects the illuminance of each of the upper shelf 13, middle shelf 14, and lower shelf 15 of the refrigerator compartment 10 (the illuminance of each space partitioned by the shelves) using the illuminance sensors 17a, 17b, and 17c. The passing object detection unit 111 then sets the detected illuminance of the upper shelf 13 as the upper reference illuminance, the illuminance of the middle shelf 14 as the middle reference illuminance, and the illuminance of the lower shelf 15 as the lower reference illuminance. The passing object detection unit 111 stores this data in the memory unit 130.

The upper reference illuminance, the middle reference illuminance, and the lower reference illuminance are the illuminances of the shelves 13, 14, and 15 when one or both of the right door 11 and the left door 12 of the refrigerator compartment 10 are open and before an object (user's hand, food, etc.) passes through the opening 10a of the refrigerator compartment 10. In addition, taking into account errors and fluctuations in the detected illuminance, the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance may be set to a slightly lower or higher illuminance than the detected illuminance of each illuminance sensor 17a, 17b, and 17c.

The upper reference illuminance, middle reference illuminance, and lower reference illuminance used in step S13 described below and the upper reference illuminance, middle reference illuminance, and lower reference illuminance used in step S14 may be set to different values. In this case, the upper reference illuminance, middle reference illuminance, and lower reference illuminance used in step S13 correspond to the first reference illuminance of the present invention, and the upper reference illuminance, middle reference illuminance, and lower reference illuminance used in step S14 correspond to the second reference illuminance of the present invention.

In the next step S12, the passing object detection unit 111 detects the illuminance of the upper shelf 13, the middle shelf 14, and the lower shelf 15 using the illuminance sensors 17a, 17b, and 17c. Then, in the next step S13, the passing object detection unit 111 determines whether there is a shelf whose detected illuminance is lower than the reference illuminance by a predetermined level or more. Specifically, the passing object detection unit 111 determines whether each of the following conditions (1) to (3) is met.

(1) The illuminance detected by the illuminance sensor 17a has changed from the "upper reference illuminance" to "upper reference illuminance - α" or lower.
(2) The illuminance detected by the illuminance sensor 17b has changed from the "medium reference illuminance" to "medium reference illuminance -α" or lower.
(3) The illuminance detected by the illuminance sensor 17c has changed from the "lower reference illuminance" to "lower reference illuminance -α" or less.

where α is a constant for setting a threshold determined by experiments, computer simulations, etc., assuming a drop in illuminance that occurs when an object (food, a user's hand, etc.) enters the upper shelf 13, the middle shelf 14, or the lower shelf 15. α may be a common value for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance, or may be set to different values individually for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance.

The passing object detection unit 111 determines that an object has passed through the opening 10a when at least one of the above (1) to (3) is met. Here, when at least one of the above (1) to (3) is met, it is assumed that the user has placed their hand inside the refrigerator compartment 10 to put food into the refrigerator compartment 10 (storage), to take food out of the refrigerator compartment 10 (removal), or to change the position of food in the refrigerator compartment 10. In this case, the passing object detection unit 111 can determine that an object (the user's hand, food being held by the user's hand, etc.) has passed through the opening 10a of the refrigerator compartment 10.

Then, the imaging time setting unit 116 sets the time when the passing object detection unit 111 determines in step S13 that at least one of the above (1) to (3) is established as the first imaging time, and proceeds to step S20. The first imaging time is set assuming that the food to be stored in the refrigerator compartment 10 passes through the opening 10a. Note that the first imaging time may be set to a time that is a predetermined time after the time when the passing object detection unit 111 determines in step S13 that at least one of the above (1) to (3) is established.

On the other hand, when none of the above (1) to (3) is satisfied, the image capture time setting unit 116 proceeds to step S14. Step S14 is a process performed by the passing object detection unit 111 and the image capture time setting unit 116. The passing object detection unit 111 determines whether there is a shelf where the detected illuminance has changed from less than or equal to "reference illuminance - β" to more than or equal to "reference illuminance". Specifically, the passing object detection unit 111 determines whether each of the following conditions (4) to (6) is satisfied.

(4) The illuminance detected by the illuminance sensor 17a has changed from less than the "upper reference illuminance-β" to more than the "upper reference illuminance."
(5) The illuminance detected by the illuminance sensor 17b has changed from less than or equal to the "middle reference illuminance-β" to more than or equal to the "middle reference illuminance."
(6) The illuminance detected by the illuminance sensor 17c has changed from less than the "lower reference illuminance-β" to more than the "lower reference illuminance."

where β is a constant for determining the threshold determined by experiments, computer simulations, etc., assuming an increase in illuminance that occurs when an object (food, a user's hand, etc.) that has entered the upper shelf 13, middle shelf 14, or lower shelf 15 passes through the opening 10a and leaves the refrigerator compartment 10. Note that β may be set to a common value for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance, or may be set to different values individually for the upper reference illuminance, the middle reference illuminance, and the lower reference illuminance.

When at least one of the above (4) to (6) is met, the passing object detection unit 111 determines that an object (food, the user's hand, etc.) that was in the refrigerator compartment 10 has left the refrigerator compartment 10 and passed through the opening 10a. Here, when at least one of the above (4) to (6) is met, it is assumed that the user has removed their hand that was in the refrigerator compartment 10 to put food into the refrigerator compartment 10 (storage), to take food out of the refrigerator compartment 10 (removal), or to change the position of food stored in the refrigerator compartment 10. In this case, the passing object detection unit 111 can determine that an object (the user's hand, food held by the user's hand, etc.) has passed through the opening 10a of the refrigerator compartment 10.

Then, the image capture time setting unit 116 sets the time when the passing object detection unit 111 determines in step S14 that at least one of the above conditions (4) to (6) is met as the second image capture time, and proceeds to step S22. The second image capture time is set assuming that the food being removed from the refrigerator compartment 10 passes through the opening 10a. Note that in step S14, the second image capture time may be set to a time that is a predetermined time after the time when the passing object detection unit 111 determines in step S14 that at least one of the above conditions (4) to (6) is met.

On the other hand, when none of the above (4) to (6) is true, the passing object detection unit 111 advances the process to step S15. In step S15, the passing object detection unit 111 determines whether or not both the right door 11 and the left door 12 of the refrigerator compartment 10 are closed, based on the detection signals of the open/close sensors 18a, 18b. Then, when both the right door 11 and the left door 12 are closed, the passing object detection unit 111 advances the process to step S15 and ends the "imaging process". On the other hand, when at least one of the right door 11 and the left door 12 is open, the passing object detection unit 111 advances the process to step S12.

Next, steps S20 to S21 and steps S22 to S23 are processes performed by the food information acquisition unit 114. In step S20, the food information acquisition unit 114 captures an image of an area including the opening 10a of the refrigerator compartment 10 with the camera 60 via the image processing unit 120. In the following step S21, the food information acquisition unit 114 stores the image data captured in step S20 in the memory unit 130 as "entry image" data, and proceeds to step S15. Note that the image may be captured multiple times, and multiple image data may be stored in the memory unit 130 as "entry image" data.

In step S22, the food information acquisition unit 114 captures an image of an area including the opening 10a of the refrigerator compartment 10 using the camera 60 via the image processing unit 120. In the following step S23, the food information acquisition unit 114 stores the image data captured in step S22 in the memory unit 130 as "image at time of leaving the warehouse" data, and proceeds to step S15. Note that the image may be captured multiple times, and multiple image data may be stored in the memory unit 130 as "image at time of leaving the warehouse" data.

By the "imaging process" shown in FIG. 7, the passing object detection unit 111 detects an object passing through the opening 10a of the refrigerator compartment 10 based on the change in illuminance of the upper shelf 13, the middle shelf 14, or the lower shelf 15 detected by the illuminance sensors 17a, 7b, and 17c.

In addition, the passing object detection unit 111 stores in the memory unit 130 an "entry image", which is an image captured near the opening 10a at the time when the passing object detection unit 111 detects an object that has passed through the opening 10a and entered one of the shelves 13, 14, or 15. In addition, the passing object detection unit 111 stores in the memory unit 130 an "exit image", which is an image captured near the opening 10a at the time when the passing object detection unit 111 detects an object that has left one of the shelves 13, 14, or 15 and passed through the opening 10a.

Note that, if an object passing through the opening 10a is detected multiple times in step S13 before it is detected in step S15 of FIG. 7 that the right door 11 and the left door 12 are closed, multiple "entry images" are captured and stored in steps S20 and S21. Then, "image analysis processing" is performed on each of the "entry images" of foods determined to have been entered by the "weight information processing" described below.

Similarly, when an object passing through the opening 10a is detected multiple times in step S14 before it is detected in step S15 of FIG. 7 that the right door 11 and the left door 12 are closed, multiple "images at the time of leaving the warehouse" are captured and stored in steps S22 and S23. Then, "image analysis processing" is performed on each of the "images at the time of leaving the warehouse" of the food determined to have been left the warehouse by the "weight information processing" described below.

[3-2. Weight information processing]
Next, the execution procedure of the "weight processing" will be described with reference to the flowchart shown in Fig. 8. The "weight information processing" is executed by the incoming/outgoing determination unit 113 and the food information acquisition unit 114.

Steps S30 to S33 and S40 in FIG. 8 are processes performed by the entry/exit determination unit 113. In step S30, the entry/exit determination unit 113 recognizes the weight of each shelf 13, 14, 15 using the storage capacity recognition unit 112 and stores the weight in the memory unit 130. In the following step S31, when the passing object detection unit 111 detects an object passing through the opening 10a of the refrigerator compartment 10 in step S13 or step S14 in FIG. 7 (when an object enters or leaves the refrigerator compartment 10), the entry/exit determination unit 113 advances the process to step S32.

In step S32, the entry/exit determination unit 113 recognizes the weight of each shelf 13, 14, 15 using the storage capacity recognition unit 112. In the next step S33, the entry/exit determination unit 113 determines whether there is a shelf where the weight recognized in step S32 has increased compared to the weight recognized in step S30. Then, if there is a shelf where the weight has increased (in this case, it can be determined that food has been entered into the refrigerator compartment 10), the entry/exit determination unit 113 proceeds to step S34.

On the other hand, if there is no shelf where the weight has increased, the entry/exit determination unit 113 proceeds from step S33 to step S40, and determines whether there is a shelf where the weight recognized in step S32 has decreased compared to the weight recognized in step S30. If there is a shelf where the weight has decreased (in this case, it can be determined that food has been removed from the refrigerator compartment 10), the entry/exit determination unit 113 proceeds to step S41. If there is no shelf where the weight has increased (in this case, it can be determined that food has not been entered into or removed from the refrigerator compartment 10), the entry/exit determination unit 113 proceeds to step S36 and ends "weight information processing".

Steps S34 to S35 and step S41 are processes performed by the food information acquisition unit 114. In step S34, the food information acquisition unit 114 estimates the storage position of the stored food from the left and right weights of each shelf 13, 14, and 15 recognized by the storage capacity recognition unit 112.

In the next step S35, the food information acquisition unit 114 records the weight and storage location of the stored food in the stored food information 132 (see FIG. 5), proceeds to step S36, and ends the "weight information processing". The stored food information 132 data is temporarily stored in the memory unit 130 in order to add or update information to the in-warehouse food list 131.

In addition, in step S41, the food information acquisition unit 114 records the weight information of the outgoing food in the outgoing food information 133 (see FIG. 5) and proceeds to step S36, terminating the "weight information processing." The data of the outgoing food information 133 is temporarily stored in the memory unit 130 in order to add or update food information to the in-warehouse food list 131.

[3-3. Image analysis processing]
Next, the execution procedure of the "image analysis process" will be described with reference to the flowcharts shown in Figures 9 and 10. The "image analysis process" is executed by the food information acquisition unit 114. Figure 9 shows a process for acquiring information from an "image at the time of entry" for food stored in the refrigerator compartment 10. Figure 10 shows a process for acquiring information from an "image at the time of exit" for food removed from the refrigerator compartment 10.

First, the "image analysis process" for the "image at time of storage" will be described with reference to FIG. 9. In step S50 of FIG. 9, the food information acquisition unit 114 executes character extraction process on the "image at time of storage" of the stored food by the image processing unit 120. Here, stored food is food that is determined to have been stored in the refrigerator compartment 10 by the storage/removal determination unit 113 in step S33 of FIG. 8. In addition, the "image at time of storage" of stored food is the "image at time of storage" captured and stored by the food information acquisition unit 114 in steps S20 and S21 of FIG. 7 for the food determined to have been stored in step S33 of FIG. 8.

In the next step S51, the food information acquisition unit 114 determines whether or not characters have been extracted from the "warehouse image" by the image processing unit 120. If characters have been extracted, the food information acquisition unit 114 proceeds to step S52, and if characters have not been extracted, the food information acquisition unit 114 proceeds to step S60.

In step S52, the food information acquisition unit 114 analyzes the text information extracted by the image processing unit 120. Then, in the following step S53, the food information acquisition unit 114 acquires information about the stored food, such as the food name, expiration date, place of origin, and content volume, from the text information. In the next step S54, the food information acquisition unit 114 records the acquired information about the stored food in the stored food information 132, and ends the "image analysis process."

In step S60, the food information acquisition unit 114 extracts an image portion of the stored food from the "stored image" by the image processing unit 120. In the following step S61, the food information acquisition unit 114 calculates the degree of match between the extracted image portion of the stored food and sample images 134 (see FIG. 5) of various foods stored in advance in the memory unit 130.

In the next step S62, the food information acquisition unit 114 estimates and acquires the food name of the sample image 134 with the highest matching rate as the food name of the stored food, and proceeds to step S54. By processing steps S60 to S62, even if characters cannot be extracted from the "image at the time of storage" of the stored food, it is possible to estimate the food name of the stored food and record it in the stored food information 132.

Next, the "image analysis process" for the "image at time of leaving the warehouse" will be described with reference to FIG. 10. In step S70 of FIG. 10, the food information acquisition unit 114 executes character extraction process on the "image at time of leaving the warehouse" of the leaving food by the image processing unit 120. Here, the leaving food is the food that is determined to have been left from the refrigerator compartment 10 by the entry/exit determination unit 113 in step S40 of FIG. 8. In addition, the "image at time of leaving the warehouse" of the leaving food is the "image at time of leaving the warehouse" captured and stored by the food information acquisition unit 114 in steps S22 and S23 of FIG. 7 for the food that is determined to have been left in step S40 of FIG. 8.

In the next step S71, the food information acquisition unit 114 determines whether or not characters have been extracted from the "image at the time of leaving the warehouse" by the image processing unit 120. If characters have been extracted, the food information acquisition unit 114 proceeds to step S72, and if characters have not been extracted, the food information acquisition unit 114 proceeds to step S80.

In step S72, the food information acquisition unit 114 analyzes the text information extracted by the image processing unit 120. Then, in the following step S73, the food information acquisition unit 114 acquires information about the shipped food, such as the food name, expiration date, place of origin, and content volume, from the text information. In the next step S74, the food information acquisition unit 114 records the acquired information about the shipped food in the shipped food information 133, and ends the "image analysis process."

In step S80, the food information acquisition unit 114 extracts an image portion of the shipped food from the "shipping image" by the image processing unit 120. In the following step S81, the food information acquisition unit 114 calculates the degree of match between the extracted image portion of the shipped food and the sample images 134 of various foods stored in advance in the memory unit 130.

In the next step S82, the food information acquisition unit 114 estimates and acquires the food name of the sample image 134 with the highest matching rate as the food name of the shipped food, and proceeds to step S74. By processing steps S80 to S82, even if characters cannot be extracted from the "image at time of shipment" of the shipped food, it is possible to estimate the food name of the shipped food and record it in the shipped food information 133.

The image analysis process in FIG. 9 is performed on the image at the time of entry when entry into the refrigerator compartment 10 is determined, and the image analysis process in FIG. 10 is performed on the image at the time of exit when exit from the refrigerator compartment 10 is determined. Therefore, when the right door 11 or the left door 12 is detected by the passing object detection unit 111 instead of food, it is not determined that an entry or exit has occurred, so the image analysis process is not performed (the captured image is invalid). This makes it possible to prevent erroneous food information from being obtained from the image portion of the right door 11 or the left door 12.

[3-4. Weight sensor correction process]
The execution procedure of the "weight sensor calibration process" for correcting the detected weights of the weight sensors 13a, 13b, 14a, 14b, 15a, and 15b will be described with reference to the flowchart shown in Fig. 11. The "weight sensor correction process" is executed by the capacity recognition unit 112. The "weight sensor correction process" is a process for correcting (calibrating) the detection errors of the weight sensors 13a, 13b, 14a, 14b, 15a, and 15b that may occur due to changes over time or the like.

In step S90 of FIG. 11, when the in/out determination unit 113 determines that food has been stored in the refrigerator compartment (YES in step S33 of FIG. 8), the storage capacity recognition unit 112 advances the process to step S91. In step S91, the storage capacity recognition unit 112 determines whether or not information on the content volume of the stored food is recorded in the stored food information 132 (processing of step S54 of FIG. 9). Then, when information on the content volume of the stored food (the content volume of the food obtained in step S53 of FIG. 9) is recorded in the stored food information 132, the storage capacity recognition unit 112 advances the process to step S92.

In step S92, the storage amount recognition unit 112 calculates the ratio between the weight and content amount of the stored food recorded in the stored food information 132. The storage amount recognition unit 112 then sets the calculated ratio as the correction coefficient for the weight sensor used to detect the weight of the stored food. For example, when the stored food is stored on the upper shelf 13 and its weight is detected by the weight sensors 13a and 13b, the storage amount recognition unit 112 sets the calculated ratio as the correction coefficient for the detected weight of the weight sensors 13a and 13b.

In the next step S93, the storage amount recognition unit 112 stores the correction coefficient data in the memory unit 130, and proceeds to step S94 to end the "weight sensor correction process". Thereafter, the storage amount recognition unit 112 treats the detected weight of the weight sensors 13a and 13b multiplied by the correction coefficient as the detected weight. Similarly, when the stored food is stored on the middle shelf 14, the storage amount recognition unit 112 sets the correction coefficient for the weight sensors 14a and 14b, and when the stored food is stored on the lower shelf 15, the storage amount recognition unit 112 sets the correction coefficient for the weight sensors 15a and 15b.

On the other hand, if no content volume information is recorded in the stored food information 132 in step S91, the content volume recognition unit 112 proceeds to step S94 and ends the "weight sensor correction process." In this case, the correction coefficient is not calculated or set.

[3-5. Food management process inside the warehouse]
12 and 13, a procedure for executing the "in-fridge food management process" for managing information on food stored in the refrigerator compartment 10 will be described. The "in-fridge food management process" is executed by the food information management unit 115.

The food information management unit 115 manages information about the foods stored in the refrigerator compartment 10 using the refrigerator food list 131 (see FIG. 5). As shown in FIG. 14, the refrigerator food list 131 records the numbers (1, 2, 3, ...) assigned to the foods stored in the refrigerator compartment 10, and information about each numbered stored food, such as the food name, weight (first, current, consumed amount), storage/removal date/time (first storage, latest storage, latest storage), stock (1: available, 0: not available), expiration date, and place of origin. The food information management unit 115 generates and updates the refrigerator food list 131 based on the stored food information 132 and the stored food information 133 recorded by the food information acquisition unit 114.

In step S100 of FIG. 12, the food information management unit 115 determines whether the entry/exit determination unit 113 has determined that food has been entered into or removed from the refrigerator compartment 10, and when entry or exit has been determined, the process proceeds to step S101. In step S101, the food information management unit 115 determines whether entry has been determined, and when entry has been determined, the process proceeds to step S102. On the other hand, when entry has not been determined (when exit has been determined), the food information management unit 115 proceeds to step S110.

In step S102, the food information management unit 115 determines whether or not food has been removed from the refrigerator compartment 10 within a predetermined time. If food has been removed from the refrigerator compartment 10 within the predetermined time, the process proceeds to step S103. If food has not been removed from the refrigerator compartment 10 within the predetermined time, the process proceeds to step S111. Note that it is possible to recognize from the food in-fridge list 131 that food has been removed from the refrigerator compartment 10 within the predetermined time. The predetermined time is set to, for example, about 10 minutes, assuming that the user will use some of the food removed from the refrigerator compartment 10 and return it to the refrigerator compartment 10 (re-storing it).

In step S103, the food information management unit 115 determines whether the matching rate between the food previously sent out and the food currently stored is equal to or greater than a predetermined value. Here, the food information management unit 115 compares the information on the previously sent out product recorded in the in-store food list 131 with the information on the currently stored product recorded in the stored food information 132, and calculates the matching rate between the previously sent out food and the food currently stored based on the matching status of the food name, expiration date, place of origin, etc.

Then, if the matching rate between the food previously shipped and the food currently stocked is equal to or greater than a predetermined value, the food information management unit 115 proceeds to step S104, and if the matching rate between the food previously shipped and the food currently stocked is less than the predetermined value, the food information management unit 115 proceeds to step S112.

Step S104 is the process performed when the matching rate between the food previously sent out and the food currently stored is equal to or greater than a predetermined value, and it can be determined that the food previously sent out and the food currently stored are the same. In step S104, the food information management unit 115 updates the current weight, most recent stored date and time, and consumed amount (difference between the weight at the time of first storing and the current weight) of the food sent out before a predetermined time in the stored food list 131 based on the information recorded in the stored food information 132, and proceeds to step S105 in FIG. 13.

Step S112 is a process carried out when the matching rate between the food that was taken out and the food that was currently stored is less than a predetermined value, and it can be determined that the food that was taken out last time is different from the food that was currently stored. In step S112, the food information management unit 115 newly registers the stored food in the in-warehouse food list 131 based on the information recorded in the stored food information 132, and proceeds to step S105 in FIG. 12.

Step S111 is a process to be performed when no food is removed from the warehouse within a predetermined time. In step S111, the food information management unit 115 newly registers the stored food in the warehouse food list 131 based on the information recorded in the stored food information 132, and proceeds to step S105 in FIG. 12.

Step S110 is the process when it is determined that food has been removed from the refrigerator compartment 10. In step S110, the food information management unit 115 updates the current weight of the food (removed food) in the stored food list 131 to 0. The food information management unit 115 also updates the amount consumed of the food in the stored food list 131, updates the latest removal time based on the information recorded in the removed food information 133, and proceeds to step 105 in FIG. 13.

In step S105 of FIG. 13, the food information management unit 115 determines whether there is any food whose current weight is 0 grams as a result of the processing in step S110. If there is any food whose current weight is 0 grams, the food information management unit 115 proceeds to processing in step S105, and if there is no food whose current weight is 0 grams, the food information management unit 115 proceeds to processing in step S120.

In step S106, the food information management unit 115 updates the information of the corresponding food in the in-fridge food list 131 (food determined to have been taken out this time) to 'out of stock' (stock 0), and proceeds to step S107. In addition, in step S120, the food information management unit 115 updates the information of the corresponding food in the in-fridge food list 131 (food determined to have been received this time) to 'in stock' (stock 1), and proceeds to step S107.

In step S107, the food information management unit 115 determines whether expiration date information based on character recognition is recorded in the stored food information 132. If expiration date information is recorded in the stored food information 132, the food information management unit 115 proceeds to step S108, and if expiration date information is not recorded, the food information management unit 115 proceeds to step S121.

In step S108, the food information management unit 115 registers the expiration date of the food currently stored in the in-warehouse food list 131 based on the expiration date information recorded in the stored food information 132, and proceeds to step S109. In addition, in step S121, the food information management unit 115 refers to the expiration date list 135 stored in the memory unit 130 to search for the general expiration date of the relevant food.

As shown in FIG. 14, the expiration date list 135 is a list that associates food names with the expiration dates of the foods, and the food information management unit 115 searches for the expiration dates associated with the food names of the currently stored foods. In the following step S122, the food information management unit 115 registers the expiration date in the data of the corresponding food in the in-store food list 131, and proceeds to step S109.

In step S109, the food information management unit 115 transmits the updated in-storage food list 131 data to the management server 520 via the communication network 510 using the wireless communication unit 140, and ends the "in-storage food management process."

The management server 520 stores the data of the in-fridge food list 131 received from the refrigerator 1 in a storage device (not shown) and transmits the consumption status of each product recognized from the in-fridge food list 131, the number of days remaining until the expiration date, etc. to the user's terminal device 400. This allows the user to check the status of the products stored in the refrigerator 1 using the terminal device 400 even when they are away from home, etc.

4. Another embodiment of the imaging process
Next, another embodiment of the "imaging process" will be described with reference to the flowchart shown in Fig. 15. The flowchart shown in Fig. 15 is executed by the food information acquisition unit 114 and the imaging time point setting unit 116. The imaging time point setting unit 116 sets the time point when a first predetermined time has elapsed since the right door 11 or the left door 12 of the refrigerator compartment 10 was opened as the first imaging time point, and sets the time point when a second predetermined time has elapsed since the right door 11 and the left door 12 of the refrigerator compartment 10 were closed as the second imaging time point.

The food information acquisition unit 114 captures images using the camera 60 at the first and second image capture times from the time when it recognizes in step S130 from the detection signals of the open/close sensors 18a, 18b that one or both of the right door 11 and the left door 12 of the refrigerator compartment 10 have switched from a closed state to an open state until it recognizes in step S137 that both the right door 11 and the left door 12 have switched from a closed state to an open state.

When the food information acquisition unit 114 recognizes in step S130 that one or both of the right door 11 and the left door 12 of the refrigerator compartment 10 have switched from a closed state to an open state, the processing proceeds to step S131. Steps S131 to S132 are processes performed by the image capture time setting unit 116.

In step S131, the image capture time point setting unit 116 starts a timer that counts the first predetermined time until the timer times out. Then, in the following step S132, when the timer times out (when the first image capture time point has arrived), the process proceeds to step S133.

Steps S133 to S138 are processes performed by the food information acquisition unit 114. In step S133, the food information acquisition unit 114 captures images multiple times using the camera 60, and in the following step S134, stores the captured image data in the memory unit 130 as "entry image" data.

In the subsequent loop of steps S135 to S137, the food information acquisition unit 114 repeatedly executes the processing of steps S135 and S136 until the opening and closing sensors 18a, 18b detect in step S137 that the right door 11 and the left door 12 are closed. In step S135, the food information acquisition unit 114 captures an image using the camera 60 and records the image capture time in the storage unit 130, and in step S136, stores the captured image data in the storage unit 130 as candidate image data for when the item is taken out of the warehouse. This causes the captured image data to be stored sequentially in the storage unit 130 in chronological order.

When the opening/closing sensors 18a, 18b detect in step S137 that the right door 11 and the left door 12 are closed, the food information acquisition unit 114 proceeds to step S138. In step S138, the food information acquisition unit 114 selects image data captured a second predetermined time before the point in time when it is detected in step S137 that both the right door 11 and the left door are closed from the candidate image data stored in the storage unit 130 and stores the image data as an image at the time of leaving the warehouse in the storage unit 130.

As another configuration for setting the first and second imaging times by the imaging time setting unit 116, the first and second imaging times may be set based on the detection status of an object by the objective sensor 80 (see Figures 3 to 5) that detects an object passing through the opening 10a. In this case, the imaging time setting unit 116 sets the first imaging time to, for example, a time when a predetermined time has elapsed from the time when the objective sensor 80 switches from a state in which an object is not detected to a state in which an object is detected, based on the time. The imaging time setting unit 116 then sets the second imaging time to, for example, a time before the time when the objective sensor 80 switches from a state in which an object is detected to a state in which an object is not detected, based on the time.

5. Other embodiments
In the above embodiment, the passing object detection unit 111 detects an object passing through the opening 10a of the refrigerator compartment 10 based on the change in illuminance in the refrigerator compartment 10 detected by the illuminance sensors 17a, 17b, and 17c. As another configuration of the passing object detection unit 111, as shown in Figs. 3 to 5, an objective sensor 80 (such as a reflective or transmissive optical sensor, ultrasonic sensor, or capacitance sensor) that includes the opening 10a in its detection range and directly detects an object passing through the opening 10a may be used. In this case, the detection range of the objective sensor 80 may be set in front of the imaging range of the camera 60 in the direction from the opening 10a toward the inside of the refrigerator compartment 10. This makes it possible to prevent failure to image the food to be stored due to a delay time from the time an object passing through the opening 10a is detected until the camera 60 performs imaging.

When the opening/closing sensor 18a detects that the left door 12 is open, and when the opening/closing sensor 18b detects that the right door 11 is open, power may be started to be supplied to the camera 60, enabling the camera 60 to take an image. With this configuration, when the left door 12 and the right door are closed, power to the camera 60 can be cut off to save electricity. Also, by starting power to the camera 60 when the right door or the left door 12 is opened, it is possible to prevent failure to capture an image of food being stored due to a delay in starting the camera 60.

Furthermore, the food information acquisition unit 114 may be configured to invalidate image data captured by the camera 60 when the opening/closing sensor 18a detects that the left door 12 has switched from an open state to a closed state, or when the opening/closing sensor 18b detects that the right door 11 has switched from an open state to a closed state, within a predetermined time from when the passing object detection unit 111 detects an object passing through the opening 10a. With this configuration, when the left door 12 or the right door 11 is closed, when the left door 12 or the right door 11 is detected by the passing object detection unit 111 and imaged by the camera 60, it is possible to prevent erroneous food information from being extracted from the image of the left door 12 or the right door 11.

In the above embodiment, the storage amount recognition unit 112 recognized the amount of food stored on each shelf 13, 14, 15 of the refrigerator compartment 10 based on the weight detected by the weight sensors 13a, 13b, 14a, 14b, 15a, 15b. As another configuration of the storage amount recognition unit 112, the amount of food stored on each shelf 13, 14, 15 of the refrigerator compartment 10 may be recognized based on the illuminance detected by the illuminance sensors 17a, 17b, 17c with the lighting units 16a, 16b, 16c, 16d, 16e, 16f turned on. Alternatively, the amount of food stored on each shelf 13, 14, 15 of the refrigerator compartment 10 may be recognized by extracting an image of the food from an image of the inside of the refrigerator compartment 10 captured by the camera 60 or a separately placed camera.

In the above embodiment, the image captured by the camera 60 is processed by the image processing unit 120, but the functions of the image processing unit 120 may be provided in the CPU 110, and the captured image may be processed by the CPU 110.

In the above embodiment, the data of the sample image 134 and the expiration date list 135 are stored in the memory unit 130, but they may also be stored in the management server 520 and downloaded from the management server 520 to the control unit 100 of the refrigerator 1 when used.

Although sample image 134 is stored in memory unit 130 or management server 520, an image recognition program that has previously learned the characteristics of each food or stored item, such as color, shape, and size, may be implemented in CPU 110, and the image portion of the stored food may be processed by the image recognition program to calculate a match rate and obtain an estimated food name of the stored food. This makes sample image 134 unnecessary, making it possible to reduce the storage capacity stored in memory unit 130 or management server 520 and also speeds up the recognition processing time for estimating the food name of the stored food. The food name estimation for outgoing food can also be processed in the same manner as above.

In the above embodiment, the in/out determination unit 113 determines whether food is in or out of the refrigerator compartment 10 of the refrigerator 1, the food information acquisition unit 114 acquires information on the in/out food, and the food information management unit 115 manages information on the food stored in the refrigerator compartment 10. The switching compartment 20, freezer compartment 40, and vegetable compartment 50 are also provided with a passing object detection unit that detects objects passing through the openings 20a, 40a, and 50a, and a storage amount recognition unit that recognizes the amount of food stored therein, so that the in/out of food can be determined. In addition, a camera that captures an area including the openings 20a, 40a, and 50a can be provided to acquire information on the in/out food from the captured images (images at the time of in/out), so that information on the food stored in the switching compartment 20, freezer compartment 40, and vegetable compartment 50 can be acquired and food can be managed.

As described above, the refrigerator of the present invention can capture images of food entering and leaving the storage compartment at the anticipated time of passing through the storage compartment, thereby more reliably obtaining information about the food entering and leaving the storage compartment, and can be used to manage the food stored in the refrigerator.

REFRIGERATOR 10 Refrigerating compartment 10a Opening 11 Right door 12 Left door 13 Upper shelf 13 Shelf 13a, 13b Weight sensor 14 Middle shelf 14a, 14b Weight sensor 15 Lower shelf 15a, 15b Weight sensor 16a, 16b, 16c, 16d, 16r, 16f Illumination unit 17a, 17b, 17c Illuminance sensor 18a, 18b Opening/closing sensor 20 Switching compartment 20a Opening 22 Opening/closing sensor 30 Ice making compartment 30a Opening 32 Opening/closing sensor 32 Step 32 Opening/closing sensor 40 Step 40 Freezer compartment 40a Opening 42 Opening/closing sensor 50 Vegetable compartment 50a Opening 52 Opening/closing sensor 60 Camera 61 Compressor 62 Cooling fan 63 Cooler 64 Condenser 100 Control unit 110 CPU
REFERENCE SIGNS LIST 111 Passing object detection unit 112 Storage capacity recognition unit 113 In-stock/out-stock determination unit 114 Food information acquisition unit 115 Food information management unit 116 Imaging time setting unit 120 Image processing unit 121 Image memory 130 Storage unit 131 In-stock food list 132 In-stock food information 133 Out-stock food information 134 Sample image 135 Expiration date list 140 Wireless communication unit 141 Date and time recognition unit 400 Terminal device 500 Gateway 510 Communication network 520 Management server

Claims (3)

A refrigerator,
A storage chamber having an opening;
A door provided in the storage chamber;
an opening/closing sensor for detecting an opening/closing state of the door;
a camera that captures an image of an imaging range including the opening;
A storage unit;
The camera captures images multiple times from the time when the opening/closing sensor detects that the door has switched from the closed state to the open state until the opening/closing sensor detects that the door has switched from the open state to the closed state, and captures images from the time when the opening/closing sensor detects that the door has switched from the closed state to the open state until the time when the opening/closing sensor detects that the door has switched from the open state to the closed state after a first image capture time has elapsed, the first image capture time being set assuming that food to be stored in the storage chamber will pass through the opening. a food information acquisition unit that repeatedly captures images using the camera, and stores a plurality of captured images in the storage unit as candidate images at the time of leaving the storage unit together with the capture time, and when the opening/closing sensor detects that the door has switched from an open state to a closed state, stores in the storage unit, as a candidate image at the time of leaving the storage unit, the candidate image at the time of leaving the storage unit, which is captured at a second capturing time point that is a second predetermined time before the time when the opening/closing sensor detects that the door has switched from an open state to a closed state, the second capturing time point being set assuming a time when the food being left from the storage room passes through the opening;
A refrigerator equipped with The refrigerator according to claim 1, wherein the food information acquisition unit stores the image captured at the first imaging time point in the memory unit as an image at the time of storage, and acquires information about the food being stored in the storage chamber and the food being removed from the storage chamber based on the image at the time of storage and the image at the time of removal. A refrigerator,
A storage chamber having an opening;
A door provided in the storage chamber;
an opening/closing sensor for detecting an opening/closing state of the door;
a camera that captures an image of an imaging range including the opening;
A storage unit;
a food information acquisition unit that performs imaging using the camera a plurality of times from when the opening/closing sensor detects that the door has switched from a closed state to an open state until the opening/closing sensor detects that the door has switched from an open state to a closed state , and stores in the storage unit an image captured at a first imaging time point when a first predetermined time has elapsed since the opening/closing sensor detected that the door has switched from a closed state to an open state, the first predetermined time point being set assuming a time point when food to be stored in the storage chamber will pass through the opening , and stores in the storage unit an image captured at a second imaging time point when a second predetermined time has elapsed since the opening/closing sensor detected that the door has switched from an open state to a closed state , the second predetermined time point being set assuming a time point when food to be removed from the storage chamber will pass through the opening, the second predetermined time point being set assuming a time point when food to be removed from the storage chamber will pass through the opening, the second predetermined time point being set assuming a time point when food to be removed from the storage chamber will pass through the opening, and acquires information about the food to be stored in the storage chamber and the food to be removed from the storage chamber based on the image captured at the entrance and the image removed.
A refrigerator equipped with

Images