JP7620875B2 - PROGRAM, TERMINAL DEVICE, AND TERMINAL DEVICE CONTROL METHOD - Google Patents

Description

This disclosure relates to a program, a terminal device, and a method for controlling a terminal device.

Patent Document 1 discloses a technology in which, when an object is detected passing through the opening of the refrigerator, a camera installed on the top of the refrigerator takes a photograph, and the refrigerator or a terminal device displays information about the food stored in the refrigerator based on the photographed image.

JP 2019-070476 A

The present disclosure provides a program, a terminal device, and a method for controlling a terminal device that allows a user to intuitively understand what foods are stored in a refrigerator.

The program of the present disclosure causes a computer of a terminal device having a display unit to function as a display control unit that displays, by the display unit, an interior image showing the interior of an empty refrigerator as viewed from the front of the refrigerator, and displays food images showing the foods in areas of the displayed interior image corresponding to storage areas of the foods stored in the refrigerator, the interior image being an image according to the model of the refrigerator used by a user of the terminal device, the interior image being divided into a plurality of the areas, the number of the areas included in the interior image being a number according to the model of the refrigerator used by the user of the terminal device, and the display control unit is configured to display a display image of a food item when it is detected that food has been stored in the refrigerator. The food images showing the foods excluding the foods that have been detected to have been removed from the warehouse are displayed, and when multiple food images are displayed in one area, the multiple food images are displayed side by side, and an upper limit number of the food images that can be displayed simultaneously is set in one area, and when the number of food images to be displayed in one area corresponding to the food storage area exceeds the upper limit, the display control unit displays in the area a number of the food images to be displayed that is less than or equal to the upper limit, and switches the food images to be displayed in the area in response to an operation on the food images displayed in the area, while maintaining the number less than or equal to the upper limit .

Further, a terminal device according to the present disclosure includes a display unit, and a display control unit that displays, by the display unit, an interior image showing the interior of an empty refrigerator as viewed from the front of the refrigerator, and displays food images showing the foods in areas of the displayed interior image corresponding to storage areas for the foods stored in the refrigerator, the interior image being an image according to the model of the refrigerator used by a user of the terminal device, the interior image being divided into a plurality of the areas, the number of areas included in the interior image being a number according to the model of the refrigerator used by the user of the terminal device, and the display control unit displays food images showing the foods in areas of the food detected to have been stored in the refrigerator, The food images showing the foods excluding the foods that have been detected to have been removed from the warehouse are displayed, and when multiple food images are displayed in one area, the multiple food images are displayed side by side, and an upper limit number of the food images that can be displayed simultaneously is set in one area, and when the number of food images to be displayed in one area corresponding to the food storage area exceeds the upper limit, the display control unit displays in the area a number of the food images to be displayed that is equal to or less than the upper limit, and switches the food images to be displayed in the area in response to an operation on the food images displayed in the area, while maintaining the number equal to or less than the upper limit.

Further, a control method for a terminal device according to the present disclosure is a control method for a terminal device having a display unit, the control method comprising: displaying, by the display unit, an interior image showing the interior of an empty refrigerator as viewed from the front of the refrigerator; the interior image being an image according to the model of the refrigerator used by a user of the terminal device; the interior image being divided into a plurality of areas ; the number of areas included in the interior image being a number according to the model of the refrigerator used by the user of the terminal device; and, in the displayed interior image, areas corresponding to storage areas of foods stored in the refrigerator are displayed as areas of the refrigerator that correspond to storage areas of foods stored in the refrigerator. food images showing the food excluding the food that has been detected to have been taken out from within the warehouse are displayed, and when multiple food images are displayed in one area, the multiple food images are displayed side by side, and an upper limit number of the food images that can be displayed simultaneously is set in one area, and when the number of the food images to be displayed in one area corresponding to the food storage area exceeds the upper limit, a number of the food images to be displayed that is equal to or less than the upper limit are displayed in the area, and the food images displayed in the area are switched while maintaining the number equal to or less than the upper limit in response to an operation on the food images displayed in the area.

In the program, terminal device, and control method for the terminal device disclosed herein, an image of the food is displayed in an area corresponding to the storage area of the food stored in the refrigerator within an image showing the interior of the refrigerator. This allows the user to intuitively grasp what kind of food is stored in the refrigerator.

FIG. 1 shows a configuration of a food management system according to a first embodiment. FIG. 1 is a side view of a camera unit installed in a refrigerator according to a first embodiment, viewed from the right side. FIG. 1 is a perspective view of a position restriction member abutting against a front edge of a main box body according to the first embodiment, as viewed from the left front side; FIG. 1 is a front view of a refrigerator and a camera unit according to a first embodiment; FIG. 1 is a plan view of a refrigerator and a camera unit according to a first embodiment, viewed from the right side. FIG. 1 is a block diagram showing a configuration of a camera unit according to a first embodiment. FIG. 1 is a block diagram showing the configuration of a terminal device and a food management server according to a first embodiment. FIG. 1 is a diagram showing an example of a refrigerator compartment management database in the first embodiment. FIG. 1 is a diagram showing an example of an image of the inside of a refrigerator compartment in the first embodiment; FIG. 1 is a diagram showing an example of a withdrawal management database according to the first embodiment. FIG. 13 is a diagram showing an example of an adjustment screen in the first embodiment; FIG. 1 shows an example of a center position recording paper according to the first embodiment. A flowchart showing the operation of the camera unit in the first embodiment. A flowchart showing the operation of a food management system in the first embodiment. FIG. 1 is a diagram for explaining an entry/exit determination image in the first embodiment; FIG. 13 is a diagram showing an example of a refrigerator compartment home screen in the first embodiment. FIG. 1 shows a procedure for inputting a memo in the first embodiment. FIG. 13 is a diagram showing an example of an accommodation list screen in the first embodiment; FIG. 13 is a diagram showing an example of a new addition list screen in the first embodiment; FIG. 1 is a diagram for explaining a procedure for adding food information in the first embodiment. FIG. 13 is a diagram showing an example of a deletion target list screen in the first embodiment; A flowchart showing the operation of a food management system in the first embodiment. FIG. 13 is a diagram showing an example of a vegetable compartment home screen in the first embodiment. FIG. 13 is a diagram showing an example of a food image list screen in the first embodiment. A flowchart showing the operation of a food management system in the first embodiment. FIG. 11 is a block diagram showing the configuration of a terminal device and a food management server according to a second embodiment.

(The knowledge and other information that formed the basis of this disclosure)
At the time when the inventors came up with the present disclosure, there was a technology in which a terminal device displayed a list of foods stored in a refrigerator. However, the inventors discovered a problem in that the user could not intuitively grasp what foods were stored in the refrigerator when the list of foods was simply displayed as in the conventional technology, and came to constitute the subject of the present disclosure in order to solve the problem.
Therefore, the present disclosure provides a program, a terminal device, and a control method for a terminal device that enable a user to intuitively grasp what type of food is stored in a refrigerator.

Hereinafter, the embodiments will be described in detail with reference to the drawings. However, in some cases, more detailed explanations than necessary may be omitted. For example, detailed explanations of already well-known matters or duplicate explanations of substantially the same configurations may be omitted.
It should be noted that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

(Embodiment 1)
First, the first embodiment will be described.
[1-1. Configuration]
[1-1-1. Food control system configuration]
FIG. 1 is a diagram showing the configuration of a food management system 1000.

The food management system 1000 is a system that manages the storage date and time of the food stored in the refrigerator 1, etc.
The refrigerator 1 corresponds to an example of a storage unit in the present disclosure. The food corresponds to an example of an article in the present disclosure.

The food management system 1000 includes a refrigerator 1. The refrigerator 1 includes a main box 10 with an opening at the front. The main box 10 is formed with a refrigerator compartment 11, an ice-making compartment 12, a fresh freezing compartment 13, a freezer compartment 14, and a vegetable compartment 15. A rotating left door 11A and a right door 11B are provided at the opening at the front of the refrigerator compartment 11. The ice-making compartment 12, the fresh freezing compartment 13, the freezer compartment 14, and the vegetable compartment 15 are each provided with drawers 12A, 13A, 14A, and 15A for storing food. The drawers 14A and 15A of the present disclosure include two cases CA, an upper case JCA and a lower case GCA.
The upper case JCA corresponds to an example of a first case in the present disclosure, and the lower case GCA corresponds to an example of a second case in the present disclosure.

In the following description, if there is no distinction between the left door 11A and the right door 11B, they will be referred to as "doors" and marked with the symbol "11C." Also, in the following description, if there is no distinction between the drawers 12A, 13A, 14A, and 15A, they will be referred to as "drawers" and marked with the symbol "16A."

Food management system 1000 includes camera unit 2. Camera unit 2 is a device that takes photographs and detects food entering or leaving refrigerator 1 from the photographed images. Camera unit 2 is provided on top surface 10B of main box 10 of refrigerator 1.
The camera unit 2 corresponds to an example of an imaging device of the present disclosure.

The food management system 1000 includes a terminal device 3. The terminal device 3 is, for example, a smartphone or a tablet terminal. An application program for managing the food in the refrigerator 1 is installed on the terminal device 3, and the terminal device 3 communicates with the food management server 4 by using the function of the application program.
In the following description, this application program is referred to as the "food management application" and is given the reference number "311."
The food management application 311 corresponds to an example of a program and an application program of the present disclosure.

In FIG. 1, a user P who is at home is indicated by a solid line, and a user P who has left home H is indicated by a dotted line. When the terminal device 3 is used by a user P who is at home, it communicates with the food management server 4 via the communication device 5 or without the communication device 5, using the function of the food management app 311. When the terminal device 3 is used by a user P who has left home H and a communication connection with the communication device 5 cannot be established, it communicates with the food management server 4 without the communication device 5, using the function of the food management app 311.

The communication device 5 is connected to a global network GN, which includes the Internet, a telephone network, and other communication networks, and communicates with the food management server 4 connected to the global network GN. The communication device 5 is an interface device for connecting the terminal device 3 to the global network GN.

The food management system 1000 includes a food management server 4. The food management server 4 is a server device that manages the food in the refrigerator 1 and is connected to the global network GN. Note that in each figure, the food management server 4 is represented by a single block, but this does not necessarily mean that the food management server 4 is composed of a single server device. For example, the food management server 4 may be composed of multiple server devices with different processing contents.

Next, we will explain the configuration of the camera unit 2, the terminal device 3, and the food management server 4.

[1-1-2. Camera unit configuration]
The camera unit 2 will now be described with reference to FIGS.
2 to 4 show the X-axis, Y-axis, and Z-axis. The X-axis, Y-axis, and Z-axis are perpendicular to each other. The Z-axis indicates the up-down direction. The X-axis and Y-axis are parallel to the horizontal direction. The X-axis indicates the left-right direction and also indicates the width direction of the refrigerator 1 and the camera unit 2. The Y-axis indicates the front-rear direction and also indicates the depth direction of the refrigerator 1 and the camera unit 2. The positive direction of the X-axis indicates the right direction. The positive direction of the Y-axis indicates the front direction. The positive direction of the Z-axis indicates the upward direction.

Figure 2 is a side view of the camera unit 2 installed in the refrigerator 1, seen from the right side. Figure 3 is a perspective view of the position restriction member 205 abutting against the front edge 10A of the main box body 10, seen from the front left side.

As shown in Figures 2 and 3, the camera unit 2 comprises a main body 201 with a box-like appearance that is installed on the top surface 10B of the main box 10. Furthermore, the camera unit 2 comprises a photographing member 202 that extends forward from the main body 201 above the front of the main body 201. The photographing member 202 comprises a refrigerator compartment camera 203 and a drawer camera 204. The refrigerator compartment camera 203 and the drawer camera 204 are each provided in front of the main body 201 at the bottom of the photographing member 202.

The main body 201 is equipped with a position restricting member 205 that restricts the position of the imaging member 202 in the front-rear direction of the refrigerator 1 by abutting against the front edge 10A of the main box 10. The position restricting member 205 is a position restricting piece that extends downward from the bottom surface 206 of the main body 201 in the installed state. The position restricting member 205 extends in the left-right direction of the main body 201 on the front surface of the main body 201, and has a mark 207 in the center of the main body 201 in the left-right direction. This mark 207 indicates the center position of the camera unit 2 in the left-right direction.

The length of the position regulating member 205 is shorter than the length from the top surface 10B of the main box 10 to the packing provided between the refrigerator 1 and the door 11C in the installed state. More specifically, the length of the position regulating member 205 is shorter than the length from the top surface 10B of the main box 10 to the door packing provided approximately parallel to the top surface 10B on the upper inner surface of the door 11C in the installed state.

The main body 201 is fixed in a position regulated by the position regulating member 205 by the built-in magnet 208. The means for fixing the main body 201 is not limited to the magnet 208, and other means such as double-sided tape may be used.

Figure 4 is a front view of the refrigerator 1 and camera unit 2 as viewed from the front side. The refrigerator 1 shown in Figure 4 has the left door 11A and the right door 11B in an open state. Figure 5 is a plan view of the refrigerator 1 and camera unit 2 as viewed from the right side. The refrigerator 1 shown in Figure 5 has the drawer 15A in an open state, i.e., pulled out. The camera unit 2 shown in Figures 4 and 5 has the mark 207 located approximately in the center of the refrigerator 1 in the left-right direction, and the position control member 205 abutting the front edge 10A.

The camera unit 2 has a refrigerator compartment camera 203 and a drawer camera 204 .
The refrigerator compartment camera 203 photographs the food entering and leaving the refrigerator compartment 11 by photographing the front of the refrigerator 1 from above to below.

For example, the shooting range of the refrigerator compartment camera 203 includes the range A1 shown in FIG. 4 when viewed from the front. Range A1 is a range that includes the door pocket 111 provided in the door 11C and the opening on the front side of the refrigerator compartment 11 when the door 11C is in the open state. Also, the shooting range of the refrigerator compartment camera 203 includes the range A3 shown in FIG. 5 when viewed from the side. Range A3 is a range that includes in the front-to-back direction from the open door 11C to the front end of the shelf 112, and also includes in the up-down direction the opening of the refrigerator compartment 11.

The drawer camera 204 photographs the food being put into and taken out of the drawer-type storage compartment by photographing the area from the top front to the bottom of the refrigerator 1.

For example, the imaging range of the drawer camera 204 includes the range A2 shown in FIG. 4 when viewed from the front. Range A2 is a range that includes the drawer 16A when viewed from the front of the refrigerator 1. Furthermore, the imaging range of the drawer camera 204 includes the range A4 shown in FIG. 5 when viewed from the side. Range A4 is a range that includes the drawer 16A when it is pulled out to the maximum in the front-to-rear direction.

The camera unit 2 has a distance measuring sensor 209 for the left door, a distance measuring sensor 210 for the right door, and a distance measuring sensor 211 for the drawer.

The left door distance sensor 209 is a distance sensor for detecting the open/closed state of the left door 11A. The left door distance sensor 209 detects the distance from the left door 11A as the open/closed state of the left door 11A.

The right door distance sensor 210 is a distance sensor for detecting the open/closed state of the right door 11B. The right door distance sensor 210 detects the distance from the right door 11B as the open/closed state of the right door 11B.

The drawer distance measuring sensor 211 is a distance measuring sensor for detecting the open/closed state of the drawer 16A. The open state of the drawer 16A refers to a state in which the drawer 16A is drawn out of the corresponding drawer-type storage chamber. The closed state of the drawer 16A refers to a state in which the drawer 16A is stored in the corresponding drawer-type storage chamber. The drawer distance measuring sensor 211 detects the distance from the drawer 16A as the open/closed state of the drawer 16A.
In this embodiment, drawer distance measuring sensor 211 detects the open/closed state of each of drawers 14A, 15A.

The camera unit 2 has a human presence sensor 212. The human presence sensor 212 outputs a detection value to the camera unit control unit 20. The human presence sensor 212 is provided at a predetermined position of the photographing member 202.
The human presence sensor 212 corresponds to an example of a "sensor" in the disclosure.

Note that the arrangement and installation positions of the sensors and cameras in Figures 4 and 5 are illustrated for the convenience of explaining the camera unit 2, and the arrangement and installation positions are not limited to those shown in Figures 4 and 5.

FIG. 6 is a block diagram showing the configuration of the camera unit 2.
The camera unit 2 includes a camera unit control unit 20 , a camera unit communication unit 21 , a sensor unit 22 , an imaging unit 23 , and a power supply unit 24 .

The camera unit control unit 20 includes a camera unit processor 220, which is a processor such as a CPU (Central Processing Unit) or MPU (Micro-processing unit), and a camera unit storage unit 230. The camera unit control unit 20 controls each part of the camera unit 2 by the camera unit processor 220 reading and executing a control program stored in the camera unit storage unit 230. The camera unit processor 220 executes the control program stored in the camera unit storage unit 230, thereby functioning as a camera unit communication control unit 221, a detection value processing unit 222, a photography control unit 223, a recording control unit 224, and a food detection unit 225.

The camera unit storage unit 230 has a memory that stores the programs executed by the camera unit processor 220 and the data processed by the camera unit processor 220. The camera unit storage unit 230 stores the control programs executed by the camera unit processor 220, the camera unit ID 4112 which is identification information for the camera unit 2, and various other data. The camera unit storage unit 230 has a non-volatile storage area. The camera unit storage unit 230 may also have a volatile storage area and constitute a work area for the camera unit processor 220.

The camera unit communication unit 21 is a communication interface equipped with a communication circuit, connector, etc. that conforms to a specific communication standard, and communicates with the food management server 4 and the terminal device 3 under the control of the camera unit control unit 20. In this embodiment, the communication standard used by the camera unit communication unit 21 is exemplified as a wireless communication standard, but it may also be a wired communication standard.

The sensor section 22 includes a left door distance sensor 209, a right door distance sensor 210, a drawer distance sensor 211, and a human presence sensor 212, and outputs detection values for each sensor to the camera unit control section 20.

The photographing section 23 includes a refrigerator compartment camera 203 and a drawer camera 204. The photographing section 23 outputs the photographing results of each of the refrigerator compartment camera 203 and the drawer camera 204 to the camera unit control section 20. The refrigerator compartment camera 203 and the drawer camera 204 capture moving images.
Hereinafter, when the refrigerator compartment camera 203 and the drawer camera 204 are not distinguished from each other, they will be referred to as "camera" and given the reference number "213".

The power supply unit 24 includes hardware such as a power supply circuit, and supplies power to each component of the camera unit 2. In this embodiment, the power cable 7 of the camera unit 2 is connected to a commercial AC power source 8, and the power supply unit 24 supplies power to each component of the camera unit 2 based on the power supplied from the external commercial AC power source 8.

The source of power supply to the power supply unit 24 is not limited to the external commercial AC power source 8, but may be the refrigerator 1 or a battery. When the source is the former, the power supply unit 24 and the refrigerator 1 are equipped with hardware such as a port that complies with a standard that allows USB power supply. When the source is the former, the USB cable of the camera unit 2 is connected to a USB port of the refrigerator 1, and the refrigerator 1 supplies power to the power supply unit 24 via the USB cable. When the source is the latter, the camera unit 2 is equipped with a battery.

As described above, the camera unit control unit 20 functions as a camera unit communication control unit 221 , a detection value processing unit 222 , a photography control unit 223 , a recording control unit 224 , and a food detection unit 225 .
The food detection unit 225 corresponds to an example of an item detection unit.

The camera unit communication control unit 221 communicates with the food management server 4 via the camera unit communication unit 21.

The detection value processing unit 222 determines whether the left door 11A, the right door 11B, and the drawers 14A and 15A are open or closed based on the detection values of the sensors output from the sensor unit 22. When the detection value processing unit 222 determines that either the left door 11A or the right door 11B is open, it outputs door open state information indicating that the door 11C is open to the recording control unit 224 and the food detection unit 225. When the detection value processing unit 222 determines that both the left door 11A and the right door 11B are closed, it outputs door closed state information indicating that the door 11C is closed to the recording control unit 224 and the food detection unit 225. When the detection value processing unit 222 determines that either the drawers 14A or 15A is open, it outputs drawer open state information indicating that either the drawers 14A or 15A is open to the recording control unit 224 and the food detection unit 225. Furthermore, if the detection value processing unit 222 determines that both drawers 14A and 15A are closed, it outputs drawer closed state information indicating that drawers 14A and 15A are closed to the recording control unit 224 and the food detection unit 225.

The detection value processing unit 222 also determines whether or not a person is present around the refrigerator 1 based on the detection value of the human presence sensor 212 output from the sensor unit 22. If the detection value processing unit 222 determines that a person is present around the refrigerator 1, it outputs presence/absence information indicating that a person is present to the photography control unit 223. If the detection value processing unit 222 determines that no person is present around the refrigerator 1, it outputs presence/absence information indicating that no person is present to the photography control unit 223.

When the detection value processing unit 222 outputs presence/absence information indicating that a person is present, the photography control unit 223 causes the refrigerator compartment camera 203 and the drawer camera 204 to start photography.When the detection value processing unit 222 outputs presence/absence information indicating that a person is not present, the photography control unit 223 causes the refrigerator compartment camera 203 and the drawer camera 204 to end photography.

When the detection value processing unit 222 outputs door open state information, the recording control unit 224 starts recording the image captured by the refrigerator compartment camera 203. When the detection value processing unit 222 outputs door closed state information, the recording control unit 224 ends the started recording of the image captured by the refrigerator compartment camera 203. When the detection value processing unit 222 outputs drawer open state information, the recording control unit 224 starts recording the image captured by the drawer camera 204. When the detection value processing unit 222 outputs drawer closed state information, the recording control unit 224 ends the started recording of the image captured by the drawer camera 204. When the recording control unit 224 ends recording, it generates a video from the start to the end of recording as one recording file 231 and stores it in the camera unit storage unit 230.

The food detection unit 225 detects food that has been put into or taken out of the refrigerator 1 based on the recording file 231 stored in the camera unit memory unit 230.

Figure 7 is a block diagram showing the configuration of the terminal device 3 and the food management server 4.

[1-1-3. Food management server configuration]
First, the configuration of the food management server 4 will be described.
The food management server 4 includes a server control unit 40 and a server communication unit 41 .

The server control unit 40 includes a server processor 400, which is a processor such as a CPU or MPU, and a server memory unit 410. The server control unit 40 controls each unit of the food management server 4 by the server processor 400 reading and executing a control program stored in the server memory unit 410. The server processor 400 functions as a server communication control unit 401 and an information processing unit 402 by executing the control program stored in the server memory unit 410.

The server storage unit 410 has a memory that stores the programs executed by the server processor 400 and the data processed by the server processor 400. The server storage unit 410 stores the control programs executed by the server processor 400, the refrigerator compartment food management database 411, the drawer food management database 412, and various other data. The server storage unit 410 has a non-volatile storage area. The server storage unit 410 has a volatile storage area and may constitute a work area for the server processor 400.

FIG. 8 is a diagram showing an example of one record stored in the refrigerator compartment food management database 411.
The refrigerator compartment food management database 411 is a database that stores various information related to the food stored in the refrigerator compartment 11.
In the following description, one record stored in the refrigerator food management database 411 is referred to as a "refrigerator food management record" and is denoted by the symbol "RR."

The refrigerator food management record RR has an account ID 4111, a camera unit ID 4112, refrigerator model information 4113, refrigerator interior image data 4114, in/out information 4115, food image data 4116, storage date and time information 4117, divided area number 4118, memo entry information 4119, memo information 4130, deletion information 4131, and deletion date and time information 4132.

Account ID 4111 is identification information that identifies the account assigned to user P who uses the food management app 311.

Refrigerator model information 4113 is information indicating the model of refrigerator 1 used by user P indicated by the associated account ID 4111 in the same record.

The refrigerator compartment interior image data 4114 is image data of an image showing the interior of the refrigerator compartment 11 of the refrigerator 1. Hereinafter, the image showing the interior of the refrigerator compartment 11 will be referred to as a "refrigerator compartment interior image" and given the symbol "RG." The refrigerator compartment interior image data 4114 indicates a refrigerator compartment interior image RG corresponding to the model indicated by the corresponding refrigerator model information 4113 in the same record.
The refrigerator compartment interior image RG corresponds to an example of an interior image.

FIG. 9 is a diagram showing an example of a refrigerator interior image RG.
The refrigerator compartment interior image RG shown in Fig. 9 includes a left door pocket image RG1, a refrigerator compartment front image RG2, and a right door pocket image RG3. In the refrigerator compartment interior image RG shown in Fig. 9, the left door pocket image RG1, the refrigerator compartment front image RG2, and the right door pocket image RG3 are arranged in this order from the left in the figure. The left door pocket image RG1 is an image showing the door pocket 111 formed in the left door 11A. The refrigerator compartment front image RG2 is an image of the refrigerator compartment 11 viewed from the front when the left door 11A and the right door 11B are in an open state. The right door pocket image RG3 is an image showing the door pocket 111 formed in the right door 11B.

The refrigerator interior image RG has an image area divided into multiple areas. Each of these areas is called a "divided area" and is given the symbol "BA". The refrigerator interior image RG shown in FIG. 9 is divided into 15 divided areas BA. Each divided area BA is assigned a different number. In the following explanation, these numbers are called divided area numbers and are given the symbol "4118". Note that FIG. 9 conveniently illustrates which divided area number is assigned to which divided area BA, and the divided area number 4118 is not displayed in the refrigerator interior image RG displayed on the refrigerator home screen GM2 described later.

Returning to the explanation of Figure 8, the inbound/outbound information 4115 is information indicating whether or not there has been any new inbound or outbound activity.

The food image data 4116 is image data of food. In this embodiment, the image represented by the food image data 4116 is an image captured by the refrigerator compartment camera 203.

Storage date and time information 4117 is information that indicates the storage date and time when the food was stored in refrigerator 1.

The refrigerator food management record RR has a division area number 4118 for each food image data 4116.

Memo input information 4119 is information indicating whether or not a note has been input.

Memo information 4130 is information that indicates the entered memo.

The deletion information 4131 is information indicating whether the associated sub-record is a sub-record to be deleted from the refrigerator food management record RR. A sub-record is a record stored in the refrigerator food management record RR, and includes food image data 4116, storage date and time information 4117, divided area number 4118, memo entry presence/absence information 4119, and memo information 4130.

The deletion date and time information 4132 is information indicating the date and time when the corresponding sub-record was targeted for deletion. When the deletion information 4131 indicates a deletion target, the information processing unit 402 described below deletes the sub-record and the deletion information 4131 and deletion date and time information 4132 associated with this sub-record from the refrigerator food management record RR at a predetermined timing.

FIG. 10 is a diagram showing one record stored in the withdrawn food management database 412. As shown in FIG.
The drawer food management database 412 is a database that stores various information related to the food in the drawer-type storage compartment. In the explanation with reference to Figures 23 to 25, the drawer food management database 412 is a database related to the vegetable compartment 15.
In the following description, one record stored in the withdrawn food management database 412 is referred to as a "drawn food management record" and is denoted by the symbol "HR."

The withdrawn food management record HR has an account ID 4111, a camera unit ID 4112, update date and time information 4121, upper case information 4122, and lower case information 4124.

Update date and time information 4121 is information that indicates the date and time when the contents of the withdrawn food management record HR were updated.

The upper case information 4122 includes upper case image data 4123, food image data 4116, memo entry information 4119, and memo information 4130.

Upper case image data 4123 is image data of an upper case image JCG, which is an image of upper case JCA. The upper case image JCG is an image captured by drawing camera 204.
The upper case image JCG corresponds to an example of a first case image of the present disclosure.

In this embodiment, the image indicated by the food image data 4116 contained in the upper case information 4122 is an image cut out from the upper case image JCG.

The lower case information 4124 includes lower case image data 4125, food image data 4116, note entry information 4119, and note information 4130.

The lower case image data 4125 is image data of a lower case image GCG, which is an image of the lower case GCA. The lower case image GCG is an image captured by the drawer camera 204.
The lower case image GCG corresponds to an example of a second case image of the present disclosure.
Hereinafter, when there is no need to distinguish between the upper case image JCG and the lower case image GCG, they will be referred to as "case images" and marked with "CG."

In this embodiment, the image indicated by the food image data 4116 contained in the lower case information 4124 is an image cut out from the lower case image GCG.

Returning to the explanation of FIG. 7, the server communication unit 41 is a communication interface that includes a configuration related to communication, such as a wireless circuit and an antenna, that conforms to a specific communication standard, and communicates with devices that connect to the global network GN according to the specific communication standard. In this embodiment, the server communication unit 41 communicates with the camera unit 2 and the terminal device 3.

As described above, the server control unit 40 functions as a server communication control unit 401 and an information processing unit 402.

The server communication control unit 401 communicates with the camera unit 2 and the terminal device 3 via the server communication unit 41.

The information processing unit 402 performs various information processing based on the information received by the server communication control unit 401 from the camera unit 2 and the terminal device 3.

[1-1-4. Configuration of terminal device]
Next, the functional configuration of the terminal device 3 will be described.
The terminal device 3 includes a terminal control unit 30 , a terminal communication unit 31 , and a touch panel 32 .
The touch panel 32 corresponds to an example of a display unit.

The terminal control unit 30 includes a terminal processor 300, which is a processor such as a CPU or an MPU, and a terminal storage unit 310. The terminal control unit 30 controls each unit of the terminal device 3 by the terminal processor 300 reading and executing a control program stored in the terminal storage unit 310. A food management application 311 is installed in the terminal device 3. The terminal processor 300 functions as an application execution unit 301 by reading and executing the food management application 311.
The terminal processor 300 corresponds to an example of a computer of the present disclosure. The application execution unit 301 corresponds to an example of a display control unit of the present disclosure.

The terminal storage unit 310 has a memory that stores programs executed by the terminal processor 300 and data processed by the terminal processor 300. The terminal storage unit 310 stores the control program executed by the terminal processor 300, the food management app 311, the account ID 4111, and various other data. The terminal storage unit 310 has a non-volatile storage area. The terminal storage unit 310 may also have a volatile storage area and constitute a work area for the terminal processor 300.

The terminal communication unit 31 is a communication interface that includes a wireless circuit, an antenna, and other components that comply with a specific communication standard, and communicates with devices connected to the global network GN in accordance with the specific communication standard. In this disclosure, a wireless communication standard is given as an example of the communication standard used by the terminal communication unit 31.

The touch panel 32 includes a display panel such as a liquid crystal display panel and a touch sensor that is layered on or integrated with the display panel.

As described above, the terminal control unit 30 functions as the application execution unit 301 .
The application execution unit 301 transmits and receives various information to and from the food management server 4 via the terminal communication unit 31 .

The app execution unit 301 also displays a user interface related to the management of food in the refrigerator 1 on the touch panel 32. In the following description, this user interface is referred to as the "app UI" and is given the reference number "320." By displaying the app UI 320 on the touch panel 32, the app execution unit 301 provides the user P with various information related to the food in the refrigerator 1, and also accepts various inputs related to the food in the refrigerator 1 from the user P.

[1-2. Operation]
Next, the operation of each part related to the food management system 1000 will be described.

[1-2-1. Installation of the camera unit]
Camera unit 2 is separate from refrigerator 1. Therefore, camera unit 2 is installed on top surface 10B of refrigerator 1 by user P. As described above, position restriction member 205 restricts the position of camera unit 2 in the front-to-rear direction of refrigerator 1, so user P adjusts the position of camera unit 2 in the left-to-right direction of refrigerator 1 when installing camera unit 2.
Below, several examples of methods for adjusting the position of the camera unit 2 in the left-right direction of the refrigerator 1 will be given.

[1-2-1-1. First position adjustment]

In the first position adjustment, the user P uses the application UI 320 .
In the first position adjustment, the application executing unit 301 causes the application UI 320 of the adjustment screen GM1 to be displayed on the touch panel 32.

FIG. 11 is a diagram showing an example of the adjustment screen GM1.
The adjustment screen GM1 is provided with a photography result display area HA1. The photography result display area HA1 is a rectangular area that displays the photography results of either the refrigerator compartment camera 203 or the drawer camera 204. The photography result display area HA1 is provided with a line L1 extending in the vertical direction at approximately the center in the horizontal direction in the figure.

The adjustment screen GM1 includes adjustment method information J1. The adjustment method information J1 prompts the user to adjust the position so that the center in the left-right direction of the refrigerator 1 shown in the shooting result display area HA1 is aligned with the line L1 provided in the shooting result display area HA1.

User P installs camera unit 2 in refrigerator 1 according to the adjustment method indicated by adjustment method information J1. This enables camera unit 2 to include in the shooting range of each of refrigerator compartment camera 203 and drawer camera 204 food being put into or taken out of refrigerator 1. Camera unit 2 also enables each distance measuring sensor to detect the opening and closing of the corresponding door 11C or drawer 16A.

In the first position adjustment, the camera unit 2 transmits the image captured by either the refrigerator camera 203 or the drawer camera 204 to the terminal device 3. In the first position adjustment, the camera unit 2 and the terminal device 3 communicate with each other via the food management server 4 or directly by short-range wireless communication, etc.

The camera unit communication control unit 221 of the camera unit 2 transmits the image capture results of either the refrigerator compartment camera 203 or the drawer camera 204 to the terminal device 3. The application execution unit 301 then displays the received image capture results in the image capture result display area HA1.

[1-2-1-2. Second position adjustment]
In the second position adjustment, user P uses the center position recording paper CY.

FIG. 12 is a diagram showing an example of the center position recording paper CY.
The central position recording paper CY is a paper on which a central mark MR2 is recorded, which indicates the central position in the left-right direction of the refrigerator 1. In addition to the central mark MR2, the central position recording paper CY also has recorded thereon a left edge mark MR1, which indicates the left edge position in the left-right direction of the refrigerator 1, and a right edge mark MR3, which indicates the right edge position in the left-right direction of the refrigerator 1.

The central position recording paper CY is packed together with the refrigerator 1 and delivered to the user P. The camera unit 2 may or may not be packed together with the central position recording paper CY and the refrigerator 1. The length of the central position recording paper CY packed together with the refrigerator 1 from the left end mark MR1 to the right end mark MR3 corresponds to the width of the refrigerator 1.

In the second position adjustment, user P uses the central position recording paper CY to determine where the central position of refrigerator 1 is in the left-right direction. For example, user P attaches central position recording paper CY to refrigerator 1 so that left end mark MR1 is the left end of refrigerator 1 and right end mark MR3 is the right end of refrigerator 1, and determines the central position of refrigerator 1 in the left-right direction using central mark MR2 as a landmark. User P then adjusts the position of camera unit 2 so that mark 207 on position restriction member 205 of camera unit 2 is in the determined central position.

The second position adjustment enables the camera unit 2 to perform various detections, similar to the first position adjustment.

[1-2-1-3. Third position adjustment]
In the third position adjustment, a mark indicating the center position in the left-right direction of refrigerator 1 is provided at a predetermined position on refrigerator 1. In the third position adjustment, mark 207 on position restriction member 205 and a mark provided on refrigerator 1 are used.

In the third position adjustment, user P adjusts the position of camera unit 2 in the left-right direction of refrigerator 1 so that the position of the mark on refrigerator 1 coincides with the position of mark 207 on position restriction member 205.

The third position adjustment enables the camera unit 2 to perform various detections, similar to the first position adjustment.

Next, the operation of food management system 1000 after camera unit 2 is installed in refrigerator 1 will be described.
[1-2-2. Recording-related operations]
First, the operations relating to recording will be described.
FIG. 13 is a flowchart FA showing the operation of the camera unit 2.

In the operation shown in FIG. 13, the sensor unit 22 outputs the detection values of each sensor to the camera unit control unit 20. In the operation shown in FIG. 13, the detection value processing unit 222 performs processing based on the various detection values received from the sensor unit 22.

The detection value processing unit 222 determines whether or not a person is present near the refrigerator 1 based on the detection value of the human presence sensor 212 (step SA1).

If the detection value processing unit 222 determines that a person is present near the refrigerator 1 (step SA1: YES), it outputs presence/absence information indicating the presence of a person to the photography control unit 223 (step SA2).

When the presence/absence information indicating the presence of a person is input from the detection value processing unit 222, the photography control unit 223 causes the refrigerator compartment camera 203 and the drawer camera 204 to start photographing (step SA3).

Next, the detection value processing unit 222 determines whether the left door 11A, the right door 11B, or the drawers 14A and 15A are open based on the detection values of the various sensors output from the sensor unit 22 (step SA4).

If the detection value processing unit 222 determines that none of the left door 11A, right door 11B, drawers 14A, and 15A are open (step SA4: NO), it performs the process of step SA16.

If the detection value processing unit 222 determines that the left door 11A, the right door 11B, or the drawers 14A and 15A are open (step SA4: YES), it determines whether the door 11C is open or whether one of the drawers 14A and 15A is open (step SA5).

If the detection value processing unit 222 determines that the door 11C is open (step SA5: door), it outputs door open state information to the recording control unit 224 (step SA6).

Next, when the door open state information is input from the detection value processing unit 222, the recording control unit 224 starts recording the images captured by the refrigerator compartment camera 203 (step SA7).

Next, the detection value processing unit 222 determines whether the door 11C is closed (step SA8).

If the detection value processing unit 222 determines that the door 11C is closed (step SA8: YES), it outputs door closed state information to the recording control unit 224 (step SA9).

Next, when the door closed state information is input from the detection value processing unit 222, the recording control unit 224 ends recording of the image captured by the refrigerator compartment camera 203 (step SA10). As a result, one recording file 231 for the refrigerator compartment camera 203 is generated in the camera unit storage unit 230.

Returning to the explanation of step SA5, if the detection value processing unit 222 determines that either drawer 14A or 15A is open (step SA5: Drawer), it outputs drawer open state information to the recording control unit 224 (step SA11).

Next, when the drawer closed state information is input from the detection value processing unit 222, the recording control unit 224 starts recording the results of the capture by the drawer camera 204 (step SA12).

Next, the detection value processing unit 222 determines whether either of the open drawers 14A, 15A has been closed (step SA13).

If the detection value processing unit 222 determines that either of the open drawers 14A, 15A has been closed (step SA13: YES), it outputs drawer closed state information to the recording control unit 224 (step SA14).

Next, when the drawer closed state information is input from the detection value processing unit 222, the recording control unit 224 ends recording of the image captured by the drawer camera 204 (step SA15). As a result, one recording file 231 of the drawer camera 204 is generated in the camera unit storage unit 230.

The detection value processing unit 222 determines whether or not a person is present near the refrigerator 1 (step SA16).

If the detection value processing unit 222 determines that a person is present near the refrigerator 1 (step SA16: YES), it performs the processing from step SA4 onwards again.

On the other hand, if the detection value processing unit 222 determines that no person is present around the refrigerator 1 (step SA16: NO), it outputs presence/absence information indicating that no person is present to the photography control unit 223 (step SA17).

When the presence/absence information indicating that no person is present is input from the detection value processing unit 222, the photography control unit 223 causes the refrigerator compartment camera 203 and the drawer camera 204 to end photography (step SA18).

[1-2-3. Food detection operations]
Next, the operation relating to the detection of food will be described.
FIG. 14 is a flowchart showing the operation of the food management system 1000.
In FIG. 14, a flowchart FB shows the operation of the camera unit 2, and a flowchart FC shows the operation of the food management server 4.

The food detection unit 225 of the camera unit 2 determines whether or not a recording file 231 has been stored in the camera unit memory unit 230 (step SB1).

When the food detection unit 225 determines that the recording file 231 is stored in the camera unit storage unit 230 (step SB1: YES), it determines whether the stored recording file 231 is a recording file 231 of the refrigerator compartment camera 203 or a recording file 231 of the drawer camera 204 (step SB2). The determination in step SB2 is made based on the information output by the detection value processing unit 222.

If the food detection unit 225 determines that the recorded file 231 is from the refrigerator compartment camera 203 (step SB2: refrigerator compartment camera), it detects food based on the stored recorded file 231 (step SB3).

In step SB3, for example, the food detection unit 225 detects food from the video recording file 231 of the refrigerator compartment camera 203 based on features such as shape and color. Data necessary for food detection is stored in a memory area accessible to the food detection unit 225.

The food detection unit 225 determines whether food has been detected from the recording file 231 of the refrigerator compartment camera 203 (step SB4).

If the food detection unit 225 determines that no food was detected from the video recording file 231 of the refrigerator compartment camera 203 (step SB4: NO), it transitions the process to step SB10.

On the other hand, if the food detection unit 225 detects food from the video recording file 231 of the refrigerator compartment camera 203 (step SB4: YES), it performs an entry/exit determination process (step SB5).

The in/out determination process is a process for determining whether the food detected in step SB3 is food that has been put into refrigerator 1 or food that has been taken out of refrigerator 1. In the in/out determination process, an in/out determination image HG is superimposed on the video image shown in the recording file 231 of the refrigerator compartment camera 203.

FIG. 15 is a diagram for explaining the warehouse entry/exit determination image HG.
The storage/return determination image HG shown in FIG. 15 is superimposed on a moving image of a user P storing an egg pack in the refrigerator 1.

The entry/exit judgment image HG has a boundary line L2.
The boundary line L2 has boundary lines L21, L22, L23, L24, and L25. The boundary line L21 is a line that extends from the left end of the entry/exit judgment image HG to the right in the upper part of the image. The boundary line L22 is a line that connects the right end of the boundary line L21 to the left end of the boundary line L23. The boundary line L23 is a line that extends to the right from the left-right position of the right end of the boundary line L21 at approximately the center in the up-down direction of the entry/exit judgment image HG. The boundary line L24 is a line that connects the right end of the boundary line L23 to the left end of the boundary line L24. The boundary line L25 is a line that extends to the right from the left-right position of the right end of the boundary line L23 to the right end of the entry/exit judgment image HG in the upper part of the image.

The image area of the entry/exit judgment image HG is divided by boundary line L2 into a concave area HG1, which is a concave area, and a convex area HG2, which is a convex area.

The food detection unit 225 performs image analysis on the moving image on which the entry/exit determination image HG is superimposed to determine whether the food detected in step SB3 moved from convex region HG2 to concave region HG1, or from concave region HG1 to convex region HG2. If the food detection unit 225 determines that the food detected in step SB3 moved from convex region HG2 to concave region HG1, it determines that the food detected in step SB3 has been entered into refrigerator 1. On the other hand, if the food detection unit 225 determines that the food detected in step SB3 has moved from concave region HG1 to convex region HG2, it determines that the food detected in step SB3 has been removed from refrigerator 1.

Returning to the explanation of the flowchart FB in FIG. 14, the food detection unit 225 performs a division area number identification process (step SB6).

The divided area number identification process is a process that identifies the divided area BA that corresponds to the storage area into which the food detected in step SB3 is stored and removed, and identifies the divided area number 4118 that is assigned to the identified divided area BA.

For example, the food detection unit 225 uses image analysis to identify the storage area of the refrigerator compartment 11 into which the food detected in step SB3 was stored or removed. Also, for example, a mark identifying the storage area is attached to the end of each shelf 112 and inside each door pocket 111, and the food detection unit 225 identifies the storage area based on the positional relationship between the mark and the detected food in the video. Next, the food detection unit 225 determines which divided area BA in the refrigerator interior image RG the identified storage area corresponds to, and identifies the divided area number 4118 assigned to the identified divided area BA. The correspondence between the storage area, divided area BA, and divided area number 4118 is stored as data in the camera unit memory unit 230.

After completing the divided area number identification process, food detection unit 225 cuts out food images for each food detected in step SB3 from video recording file 231 of refrigerator compartment camera 203 (step SB7). Hereinafter, the food images cut out from the photographing results of camera 213 are referred to as "photographed food images" and are assigned the symbol "FG".
The photographed food image FG corresponds to an example of a food image of the present disclosure.

Next, the food detection unit 225 generates the first update information (step SB8).

The first update information generated in step SB 8 includes the camera unit ID 4112 of the camera unit storage section 230 .
The first update information generated in step SB8 also includes date and time information indicating the date and time when the video recording file 231 of the refrigerator compartment camera 203 was generated.
In addition, the first update information generated in step SB8 includes, for each food detected in step SB3, food image data 4116 indicating the photographed food image FG cut out in step SB7, in-stock/out-stock determination result information indicating the determination result of the in-stock/out-stock determination process, and the divided area number 4118 identified in the divided area number identification process.

Next, the camera unit communication control unit 221 transmits the first update information generated by the food detection unit 225 to the food management server 4 (step SB9).

The food detection unit 225 deletes the recording file 231 from the camera unit memory unit 230 (step SB10).

Returning to the explanation of step SB2, if the food detection unit 225 determines that the file is the recording file 231 of the drawer camera 204 (step SB2: drawer camera), it determines whether at least one of the upper case JCA and the lower case GCA shown in the video image of the recording file 231 of the drawer camera 204 has been pulled out by a predetermined threshold or more (step SB11).

For example, the food detection unit 225 distinguishes the cases CA in the moving image based on feature quantities such as shape and color. Then, the food detection unit 225 determines whether the upper case JCA and the lower case GCA have been pulled out from the refrigerator 1 by a predetermined threshold or more through image analysis of the video recording file 231.
Also, for example, a mark for identifying case CA is provided at a predetermined position on each of upper case JCA and lower case GCA, and food detection unit 225 distinguishes case CA in the moving image by reading this mark from the moving image. Then, food detection unit 225 calculates the movement distance of this mark in the moving image to determine whether or not the food has been pulled out of refrigerator 1 by a predetermined threshold or more.

If the food detection unit 225 determines that neither the upper case JCA nor the lower case GCA has been pulled out to a predetermined threshold value or more (step SB11: NO), it performs the process of step SB18.

On the other hand, if the food detection unit 225 determines that at least one of the upper case JCA and the lower case GCA has been pulled out by more than the predetermined threshold (step SB11: YES), it extracts a case image CG from the recording file 231 (step SB12).

In step SB12, the food detection unit 225 cuts out an upper case image JCG from the recording file 231 if only the upper case JCA has been pulled out by a predetermined threshold or more.
In step SB12, the food detection unit 225 cuts out a lower case image GCG from the video recording file 231 if only the lower case GCA is extracted by a predetermined threshold value or more.
In step SB12, the food detection unit 225 cuts out an upper case image JCG and a lower case image GCG from the recording file 231 if the upper case JCA and the lower case GCA are pulled out by a predetermined threshold value or more.

Next, the food detection unit 225 detects food from the cut-out case image CG (step SB13).

Next, the food detection unit 225 determines whether food has been detected from the case image CG (step SB14).

If the food detection unit 225 determines that food cannot be detected (step SB14: NO), it performs the process of step SB16.

On the other hand, if the food detection unit 225 determines that food has been detected (step SB14: YES), it cuts out a photographed food image FG for each detected food from the case image CG (step SB15).

Next, the food detection unit 225 generates second update information (step SB16).

If only the upper case image JCG is cut out, the second update information generated in step SB16 includes the camera unit ID 4112 in the camera unit memory 230, upper case image data 4123 indicating the cut-out upper case image JCG, and food image data 4116 indicating the photographed food image FG cut out from the upper case image JCG. Note that if a negative judgment is made in step SB14, this second update information does not include the food image data 4116.

In addition, if only the lower case image GCG is cut out, the second update information generated in step SB16 includes the camera unit ID 4112 of the camera unit storage section 230, lower case image data 4125 indicating the cut-out lower case image GCG, and food image data 4116 indicating the photographed food image FG cut out from the lower case image GCG. Note that if a negative judgment is made in step SB14, this second update information does not include the food image data 4116.

When the upper case image JCG and the lower case image GCG are cut out, the second update information generated in step SB16 includes the camera unit ID 4112 of the camera unit storage unit 230, the upper case image data group, and the lower case image data group. The upper case image data group includes upper case image data 4123 indicating the cut-out upper case image JCG, and food image data 4116 indicating the photographed food image FG cut out from the upper case image JCG. The lower case image data group includes lower case image data 4125 indicating the cut-out lower case image GCG, and food image data 4116 indicating the photographed food image FG cut out from the lower case image GCG. Note that when a negative determination is made in step SB14, the second update information does not include the food image data 4116.

Next, the camera unit communication control unit 221 transmits the second update information generated by the food detection unit 225 to the food management server 4 (step SB17).

Next, the food detection unit 225 deletes the recording file 231 from the camera unit memory unit 230 (step SB18).

As shown in flowchart FC, the server communication control unit 401 of the food management server 4 determines whether the first update information or the second update information has been received (step SC1).

If the server communication control unit 401 determines that update information has been received (step SC1: YES), the information processing unit 402 updates the database based on the update information received by the server communication control unit 401 (step SC2).

Step SC2 will now be described in detail.
When the update information received by the server communication control unit 401 is the first update information, the information processing unit 402 identifies the refrigerator food management record RR having the camera unit ID 4112 included in the first update information. Then, the information processing unit 402 updates the in/out presence information 4115 of the identified refrigerator food management record RR to information indicating "present."

Next, if the inbound/outbound determination result information included in the first update information indicates inbound, the information processing unit 402 generates a sub-record, associates blank deletion information 4131 and deletion date and time information 4132 with the generated sub-record, and stores the sub-record in the identified refrigerator food management record RR. The sub-record generated here includes food image data 4116 included in the first update information, storage date and time information 4117 indicating the same date and time as the date and time information included in the first update information, divided area number 4118 included in the first update information, blank memo entry presence/absence information 4119, and blank memo information 4130.

Furthermore, when the inbound/outbound determination result information included in the first update information indicates outbound, the information processing unit 402 identifies a sub-record from the refrigerator food management record RR that includes food image data 4116 indicating a photographed food image FG that has a high degree of match with the photographed food image FG indicated by the food image data 4116 included in the first update information. This identification is performed for each food image data 4116 included in the first update information. The information processing unit 402 then updates the deletion information 4131 corresponding to the identified sub-record to information indicating that it is to be deleted. Furthermore, the information processing unit 402 updates the deletion date and time information 4132 corresponding to the identified sub-record to information indicating the current date and time as the deletion date and time.

Step SC2 will now be described in more detail.
When the update information received by the server communication control unit 401 is the second update information, the information processing unit 402 identifies the withdrawn food management record HR having the camera unit ID 4112 included in the second update information.

If the second update information is for the case where only the upper case image JCG has been extracted, the information processing unit 402 updates the upper case image data 4123 and food image data 4116 contained in the upper case information 4122 of the identified drawer food management record HR to the data contained in the second update information.
If the second update information is for the case where only the lower case image GCG has been cut out, the information processing unit 402 updates the lower case image data 4125 and food image data 4116 contained in the lower case information 4124 of the identified drawer food management record HR to the data contained in the second update information.
In the second update information when the upper case image JCG and the lower case image GCG have been extracted, the information processing unit 402 appropriately updates the upper case image data 4123 and food image data 4116 held in the upper case information 4122 of the identified drawer food management record HR to data included in the upper case image data group. Also, the information processing unit 402 updates the lower case image data 4125 and food image data 4116 held in the lower case information 4124 of the identified drawer food management record HR to data included in the lower case image data group.

The information processing unit 402 also updates the update date and time information 4121 of the identified withdrawn food management record HR to information indicating the date and time when the withdrawn food management record HR was updated.

[1-2-4. Application UI-related operations]
Next, the operation related to the application UI 320 will be described.
In this embodiment, the application UI 320 displays a screen related to the refrigerator compartment 11 and a screen related to the drawer-type storage compartment.

[1-2-4-1. Refrigerator-related screens]

Screens related to the refrigerator compartment 11 include the refrigerator compartment home screen GM2, the refrigerator compartment list screen GM3, and the food information addition list screen GM4.

Figure 16 shows an example of the refrigerator compartment home screen GM2.

The refrigerator compartment home screen GM2 has a food image display area HA2.
The food image display area HA2 is an area for displaying the photographed food image FG. The food image display area HA2 displays the refrigerator compartment interior image RG, and displays the photographed food image FG superimposed on the refrigerator compartment interior image RG. Each photographed food image FG is displayed superimposed on a divided area BA indicated by a divided area number 4118 corresponding to the food image data 4116 in the refrigerator compartment interior image RG.

The food image display area HA2 in FIG. 16 displays the refrigerator compartment interior image RG shown in FIG. 9. For example, in the refrigerator compartment food management record RR, the food image data 4116 showing the photographed food image FG1 in FIG. 16 is associated with the divided area number 4118 of "4". In this case, the photographed food image FG1 is displayed superimposed on the divided area BA to which the divided area number 4118 of "4" is assigned.

The default food image display area HA2 displays the refrigerator compartment interior image RG with the left door pocket image RG1 and the right door pocket image RG3 cut off. The food image display area HA2 accepts scrolling operations in the left and right directions in the figure from the user P. The food image display area HA2 changes the display range of the refrigerator compartment interior image RG depending on the scroll amount and scroll direction of the accepted scroll operation.

For example, suppose the scroll direction is to the right and the scroll amount is such that the left edge of the left door pocket image RG1 is displayed. In this case, the food image display area HA2 displays the refrigerator compartment interior image RG with the right door pocket image RG3 completely cut off. Note that the photographed food image FG and the memo input icon MIC move in accordance with changes in the display range of the refrigerator compartment interior image RG, in other words, in conjunction with the movement of the superimposed divided area BA. The memo input icon MIC will be described later.

When multiple photographed food images FG are displayed in one divided area BA, the food image display area HA2 displays them side by side in the left-right direction of the figure. The food image display area HA2 can display up to five photographed food images FG side by side simultaneously in one divided area BA. The food image display area HA2 accepts a scroll operation from the user P when there are six or more photographed food images FG to be displayed in one divided area BA. When a scroll operation is accepted, the one divided area BA displays the sixth and subsequent photographed food images FG. When there are six or more photographed food images FG to be displayed in one divided area BA and a scroll operation in the left-right direction is accepted, the food image display area HA2 displays the sixth and subsequent photographed food images FG according to the amount of scroll while maintaining the maximum number that can be displayed side by side. At this time, the food image display area HA2 hides the first through fifth food images FG that were displayed, or displays the first and subsequent food images FG after the sixth and subsequent food images FG, depending on the amount of scrolling and the number of photographed food images FG to be displayed. Note that the memo input icon MIC moves in accordance with the movement of the associated photographed food image FG. In other words, when the associated photographed food image FG is hidden by the scrolling operation, the memo input icon MIC is also hidden.

Each photographed food image FG displayed in the food image display area HA2 accepts the input of notes. Notes are information that the user P wants to leave about the food in the refrigerator 1.

Figure 17 shows the steps for entering a note.

On the refrigerator compartment home screen GM2, the user P performs a predetermined touch operation, such as a long press operation, on the photographed food image FG on which the user wants to enter a memo. When the application execution unit 301 receives a predetermined touch operation on one photographed food image FG, it displays a memo edit image MHG in association with the photographed food image FG on which the predetermined touch operation was performed.

The memo editing image MHG has a memo input area MNA, a memo save button B1, a memo delete button B2, and a food delete button B3.

The memo input area MNA is an area for inputting a memo. When the memo input area MNA is touched, the application execution unit 301 displays a software keyboard on the touch panel 32. The application execution unit 301 then inputs a memo corresponding to the operation on the software keyboard into the memo input area MNA.

The memo save button B1 is a software button for saving a memo input in the memo input area MNA. The memo save button B1 cannot be touched if no memo is input in the memo input area MNA. When the memo save button B1 is touched, the application execution unit 301 displays the photographed food image FG to which the memo is input in a display mode different from the display mode of the photographed food image FG to which the memo is not input on the screen related to the refrigerator compartment 11. In this embodiment, the application execution unit 301 displays the frame of the photographed food image FG to which the memo is input in a different color, and displays the photographed food image FG to which the memo is input in association with a memo input icon MIC, which is an icon indicating that a memo has been input. The memo input icon MIC can be touched. When the memo input icon MIC is touched, the input memo is displayed in the application UI 320.
The memo input icon MIC corresponds to an example of a memo input image of the present disclosure.

The memo erase button B2 is a button for erasing the input memo. When the memo erase button B2 is touched, the memo input in the photographed food image FG is erased.

The food delete button B3 is a button for deleting the photographed food image FG from the food image display area HA2.

Returning to the explanation of the refrigerator compartment home screen GM2 in FIG. 16, the food image display area HA2 has a list display button B4. The list display button B4 is a software button that can be operated by touch.

When the list display button B4 is touched, the app UI320 transitions to the refrigerator compartment list screen GM3.

The refrigerator compartment list screen GM3 includes a storage list screen GM31, a new addition list screen GM32, and a deletion target list screen GM33.

The accommodation list screen GM31 will be described.
FIG. 18 is a diagram showing the accommodation list screen GM31.

The storage list screen GM31 has a stored food information display area HA3. The stored food information display area HA3 displays a list of first food information J21 for each food stored in the refrigerator compartment 11. The first food information J21 includes a photographed food image FG, storage date and time information 4117, and divided area information J3. The divided area information J3 is an image RG of the interior of the refrigerator compartment in which the divided area BA is displayed in color.

When a note has been entered into the photographed food image FG of the first food information J21, the frame is displayed in a different color and a note input icon MIC is associated with it. Note that in the contained food information display area HA3, scrolling is possible in the vertical direction of the figure. By scrolling in the contained food information display area HA3, user P can display the first food information J21 that cannot be displayed in the contained food information display area HA3.

The storage list screen GM31 has a first switching button B5 and a second switching button B6.

The first switching button B5 is a software button for switching the order of the first food information J21 displayed in the stored food information display area HA3 to the next order.
That is, when the first switching button B5 is touched, the stored food information display area HA3 displays the first food information J21 having the photographed food image FG to which a memo has been inputted higher than the first food information J21 having the photographed food image FG to which a memo has not been inputted. Displaying higher means displaying in a manner that reduces the amount of scrolling when displaying. In addition, when there are multiple pieces of first food information J21 having photographed food images FG to which a memo has been inputted, the stored food information display area HA3 displays the food information J2 having the oldest storage date and time indicated by the storage date and time information 4117 higher. In addition, when there are multiple pieces of first food information J21 having photographed food images FG to which a memo has not been inputted, the stored food information display area HA3 displays the first food information J21 having the oldest storage date and time indicated by the storage date and time information 4117 higher.
The accommodation list screen GM31 on the left side of FIG. 18 shows the accommodation list screen GM31 when the first switching button B5 is touched.

The second switching button B6 is a software button for switching the order of the first food information J21 displayed in the stored food information display area HA3 to the next order.
That is, when the second switching button B6 is touched, the contained food information display area HA3 displays the first food information J21 in each divided area BA. The order of the divided areas BA may be an order predetermined by the user P. The contained food information display area HA3 displays the first food information J21 in one divided area BA in the same order as when the first switching button B5 is touched.
The accommodation list screen GM31 on the right side of FIG. 18 shows the accommodation list screen GM31 when the second switching button B6 is touched.

The accommodation list screen GM31 has an accommodation list display button B7, a new addition list display button B8, and a deletion target list display button B9.

The accommodation list display button B7 is a software button for switching the screen displayed by the application UI320 to the accommodation list screen GM31.
The new addition list display button B8 is a software button for switching the screen displayed by the application UI320 to a new addition list screen GM32.
The deletion target list display button B9 is a software button for switching the screen displayed by the application UI320 to a deletion target list screen GM33.

FIG. 19 is a diagram showing an example of the new addition list screen GM32.
The newly added food list screen GM32 has a newly added food information display area HA4. The newly added food information display area HA4 displays a list of first food information J21 related to foods newly stored in the refrigerator compartment 11. Note that foods newly stored in the refrigerator compartment 11 refer to foods stored within a predetermined period (e.g., three days) from the current date and time. The newly added food information display area HA4 displays the first food information J21 of foods corresponding to newly stored foods in order of the most recent storage date and time indicated by the storage date and time information 4117.

Each of the first food information J21 displayed in the newly added food information display area HA4 accepts a predetermined touch operation on the photographed food image FG. On the newly added food list screen GM32, the first edited image HNG1 is displayed in association with the photographed food image FG that has been subjected to the predetermined touch operation.

The first edited image HNG1 has a food delete button B10, an area correction button B11, and a back button B12.
The food deletion button B10 is a software button for, when touched, marking the food indicated by the photographed food image FG associated with the first edited image HNG1 as a target for deletion from the refrigerator compartment home screen GM2.
The area correction button B11 is a software button for correcting the divided area BA indicated by the divided area information J3 when touched.
The back button B12 is a software button for stopping the display of the first edited image HNG1.

The newly added food information display area HA4 has an add food information button B13 at the end of the list of first food information J21. The add food information button B13 is a software button for manually adding new first food information J21.
The procedure for adding the first food information J21 will be described with reference to FIG.

Figure 20 is a diagram explaining the procedure for adding the first food information J21.

When the Add Food Information button B13 is touched, the Add Food Information list screen GM4 is displayed in the app UI320. The Add Food Information list screen GM4 has a plurality of Add buttons B14, which are software buttons. The Add Food Information list screen GM4 also has a blank image BG and a food name input area SNA for inputting a food name. When the blank image BG is touched, an image appears in the blank image BG showing the food name input in the food name input area SNA.

When the Add button B14 is touched on the food information addition list screen GM4, a refrigerator compartment interior image RG is displayed on the app UI320. Here, the refrigerator compartment interior image RG displayed is an image in which each divided area BA is assigned a divided area number 4118, and each divided area BA can be selected by touching. When a divided area BA is selected on this refrigerator compartment interior image RG, the next first food information J21 is added on the app UI320.

That is, the first food information J21 to be added is an image of the food corresponding to the touched add button B14, storage date and time information 4117 indicating the date and time when the first food information J21 was added as the storage date and time, and divided area information J3 indicating the selected divided area BA. Note that the image of the food indicated by the first food information J21 to be added may be an icon. In this embodiment, it is assumed that the image superimposed on the refrigerator compartment interior image RG on the refrigerator compartment home screen GM2 is a shadow food image FG, but when the first food information J21 is added, an icon is displayed on the refrigerator compartment interior image RG together with the photographed food image FG.

The procedure for adding the first food information J21 may include a procedure in which the user P inputs the storage date and time. In this case, the added first food information J21 includes storage date and time information 4117 indicating the storage date and time input by the user P.

Returning to the explanation of FIG. 19, the new addition list screen GM32 has a storage list display button B7, a new addition list display button B8, and a deletion target list display button B9.

Figure 21 shows an example of the deletion target list screen GM33.

The deletion target list screen GM33 has a deletion target food information display area HA5. The deletion target food information display area HA5 displays a list of second food information J22 for each food to be deleted. Note that foods to be deleted refer to foods to be deleted from the refrigerator compartment home screen GM2 and foods that have been determined to have been removed from the refrigerator compartment 11. The second food information J22 includes the photographed food image FG, deletion date and time information 4132, and divided area information J3. The deletion target list screen GM33 displays the second food information J22 in order of most recent deletion date and time.

For each of the second food information J22 displayed in the deletion target food information display area HA5, the photographed food image FG receives a predetermined touch operation. On the deletion target list screen GM33, the second edited image HNG2 is displayed in association with the photographed food image FG that has been subjected to the predetermined touch operation.

The second edited image HNG2 has a delete cancel button B15 and a back button B16.
The cancel deletion button B15 is a software button for making the photographed food image FG associated with the second edited image HNG2 the image to be displayed on the refrigerator compartment home screen GM2.
The back button B16 is a software button for stopping the display of the second edited image HNG2.

[1-2-4-2. Operation of food control system]
Next, the operation of the food management system 1000 when displaying a screen related to the refrigerator compartment 11 will be described.

Figure 22 is a flowchart showing the operation of the food management system 1000. In Figure 22, flowchart FD shows the operation of the terminal device 3, and flowchart FE shows the operation of the food management server 4.

The application execution unit 301 of the terminal device 3 determines whether to request the refrigerator food management record RR from the food management server 4 (step SD1).

For example, when the application executing unit 301 receives a touch operation instructing the application UI 320 to display the refrigerator compartment home screen GM2, the application executing unit 301 determines in step SD1 as affirmative.
Also, for example, when a touch operation instructing to update the display is performed on the refrigerator compartment home screen GM2, the application executing unit 301 makes an affirmative determination in step SD1.

When the application execution unit 301 determines that the refrigerator management record RR is requested (step SD1: YES), it transmits first record request information requesting the refrigerator management record RR to the food management server 4 (step SD2). The first record request information transmitted in step SD2 includes the account ID 4111 stored in the device storage unit 310.

As shown in flowchart FF, the server communication control unit 401 of the food management server 4 receives the first record request information via the server communication unit 41 (step SE1).

Next, the information processing unit 402 identifies the refrigerator food management record RR that includes the account ID 4111 included in the first record request information received by the server communication control unit 401 from the refrigerator food management database 411 (step SE2).

Then, the server communication control unit 401 transmits the refrigerator food management record RR identified by the information processing unit 402 to the terminal device 3 (step SE3). The information processing unit 402 updates the in-stock/out-stock presence/absence information 4115 of the transmitted refrigerator food management record RR to information indicating "absent."

Referring to the flowchart FD, when the application execution unit 301 of the terminal device 3 receives the refrigerator food management record RR from the food management server 4 (step SD3), it generates a screen related to the refrigerator compartment 11 based on this refrigerator food management record RR (step SD4).

Step SD4 will now be described in detail.
The application executing unit 301 generates a refrigerator home screen GM2 that displays, in the food image display area HA2, a refrigerator interior image RG indicated by the refrigerator interior image data 4114 included in the refrigerator food management record RR.
In addition, the application execution unit 301 generates a refrigerator compartment home screen GM2 in which the photographed food image FG indicated by the food management image data 4116 is superimposed on the divided area BA indicated by the divided area number 4118 for each sub-record.
Furthermore, when the memo input presence/absence information 4119 indicates that a memo is present, the application executing unit 301 generates a refrigerator home screen GM2 that displays a memo input icon MIC associated with the photographed food image FG for each sub-record.

Step SD4 will now be described in detail.
The application executing unit 301 generates the first food information J21 for each food image data 4116. The first food information J21 includes the photographed food image FG indicated by the food image data 4116, the storage date and time information 4117 corresponding to the food image data 4116, and the divided area information J3 indicated by the divided area number 4118 in color. The application executing unit 301 then generates a storage list screen GM31 that displays a list of the first food information J21 related to the sub-records that correspond to the deletion information 4131 that does not indicate the deletion target. The application executing unit 301 also generates a new addition list screen GM32 that displays a list of the first food information J21 displayed by the storage list screen GM31, the first food information J21 whose storage date and time information 4117 indicates a date and time within a predetermined period from the current time.

Step SD4 will now be described in detail.
The application executing unit 301 generates second food information J22 for each food image data 4116. The second food information J22 includes the photographed food image FG indicated by the food image data 4116, deletion date and time information 4132 corresponding to the food image data 4116, and divided area information J3 indicated by the divided area number 4118 in color. The application executing unit 301 then generates a deletion target list screen GM33 that displays a list of the generated second food information J22.

Returning to the explanation of the flowchart FD, the application execution unit 301 displays a screen related to the refrigerator compartment 11 in response to the touch operation (step SD5). If the input/output presence/absence information 4115 of the refrigerator compartment food management record RR received in step SD3 indicates "present," when the refrigerator compartment home screen GM2 is displayed for the first time in step SD5, the application execution unit 301 displays a pop-up on the refrigerator compartment home screen GM2 indicating that new input/output has occurred.

Next, we will explain the operation of the food management system 1000 when operations are performed on the screen related to the refrigerator compartment 11.

When the memo save button B1 is touched, the application execution unit 301 transmits information to the food management server 4 to update the sub-record related to the photographed food image FG that is the target of the operation. This information includes memo input presence/absence information 4119 indicating the presence of a memo, and memo information 4130 indicating the memo that has been input. The food management server 4 updates the sub-record appropriately based on the received information.

When the memo erase button B2 is touched, the application execution unit 301 sends information to the food management server 4 to update the sub-record related to the photographed food image FG that is the target of the operation. This information includes memo entry presence/absence information 4119, which indicates that no memo has been entered. The food management server 4 updates the sub-record appropriately based on the received information.

When the food deletion buttons B3, B10 are touched, the application execution unit 301 transmits information to the food management server 4 for updating the sub-record related to the photographed food image FG to be operated. This information includes deletion information 4131 indicating the item to be deleted. The food management server 4 updates the sub-record appropriately based on the received information.

When the area correction button B11 is touched to correct the divided area BA, the application execution unit 301 transmits information to the food management server 4 to update the sub-record related to the photographed food image FG that is the subject of the operation. This information includes the divided area number 4118 of the corrected divided area BA. The food management server 4 updates the sub-record appropriately based on the received information.

When the first food information J21 is added using the addition procedure shown in FIG. 20, the application execution unit 301 transmits information for adding a sub-record related to the added first food information J21 to the food management server 4. The food management server 4 updates the sub-record appropriately based on the received information.

When the cancel deletion button B15 is touched, the application execution unit 301 transmits information to the food management server 4 to update the sub-record related to the photographed food image FG that is the target of the operation. This information includes deletion information 4131 indicating a blank. The food management server 4 updates the sub-record appropriately based on the received information.

[1-2-4-3. Drawer-type Containment Room Screen]
Next, a screen related to the drawer-type storage compartment will be described. In the description using Figures 23 to 25, the vegetable compartment 15 is exemplified as a drawer-type storage compartment. In this description, the database shown in Figure 10 is a database that stores various information related to the food in the vegetable compartment 15.

Screens related to the vegetable compartment 15 include the vegetable compartment home screen GM5 and the food image list screen GM6.

Figure 23 shows an example of the vegetable compartment home screen GM5.

The vegetable compartment home screen GM5 has a case image display area HA6.
The case image display area HA6 is an area for displaying a case image CG. The case image display area HA6 displays an upper case image JCG superimposed on a lower case image GCG. In the default case image display area HA6, the upper case image JCG is superimposed on the entire image area of the lower case image GCG, and only the upper case image JCG is displayed.

The case image display area HA6 accepts scrolling operations in the up and down directions of the figure. The case image display area HA6 changes the display range of the upper case image JCG depending on the scroll amount and scroll direction of the accepted scroll operation. For example, if the scroll direction of the accepted scroll operation is upward, the case image display area HA6 displays a partially cut-off upper case image JCG. In the case image display area HA6, when the upper case image JCG moves upward, the lower case image GCG on which the upper case image JCG was superimposed is displayed.

Each case image CG displayed in the case image display area HA6 accepts the input of notes. The procedure for inputting notes is the same as in the case of the refrigerator compartment 11, except that the image operated by the user P is either a photographed food image FG or a case image CG. As shown in FIG. 23, the note input icon MIC associated with the upper case image JCG moves in accordance with the movement of the upper case image JCG.

The vegetable compartment home screen GM5 has a list display button B17. The list display button B17 is a software button that can be operated by touch.

When the list display button B17 is touched, the food image list screen GM6 is displayed in the app UI320.

The vegetable compartment home screen GM5 may have a switch button for switching the case image CG displayed in the case image display area HA6 to the upper case image JCG or the lower case image GCG. In this configuration, the application execution unit 301 switches the case image CG to be displayed to the upper case image JCG or the lower case image GCG in response to an operation on this switch button.

Figure 24 shows an example of the food image list screen GM6.

The food image list screen GM6 has update date and time information 4121. The food image list screen GM6 in FIG. 24 has update date and time information 4121 that indicates the update date and time of "12/1 10:05".

The food image list screen GM6 displays a list of photographed food images FG for each case. The photographed food image FG displayed in association with the upper case JCA item is an image cropped from the upper case image JCG. The photographed food image FG displayed in association with the lower case GCA item is an image cropped from the lower case image GCG.

[1-2-4-5. Operation of food control system]
Next, the operation of the food management system 1000 when displaying a screen related to the vegetable compartment 15 will be described.

Figure 25 is a flowchart showing the operation of the food management system 1000. In Figure 11, flowchart FF shows the operation of the terminal device 3, and flowchart FJ shows the operation of the food management server 4.

The application execution unit 301 of the terminal device 3 determines whether to request the withdrawal management record HR from the food management server 4 (step SF1).

For example, when the application executing unit 301 receives a touch operation instructing the application UI 320 to display the vegetable compartment home screen GM5, the application executing unit 301 makes a positive determination in step SF1.
Also, for example, when a touch operation instructing to update the display is performed on the vegetable compartment home screen GM5, the application executing unit 301 makes an affirmative determination in step SF1.

When the application execution unit 301 determines that the withdrawal management record RR is requested (step SF1: YES), it transmits second record request information requesting the withdrawal management record HR to the food management server 4 (step SF2). The second record request information transmitted in step SF2 includes the account ID 4111 stored in the device storage unit 310.

Referring to flowchart FJ, the server communication control unit 401 of the food management server 4 receives the second record request information (step SJ1).

Next, the information processing unit 402 identifies the withdrawal management record HR that includes the account ID 4111 included in the second record request information received by the server communication control unit 401 from the withdrawal food management database 412 (step SJ2).

Then, the server communication control unit 401 transmits the withdrawal management record HR identified by the information processing unit 402 to the terminal device 3 (step SJ3).

Referring to flowchart FJ, when the application execution unit 301 of the terminal device 3 receives the withdrawn food management record HR from the food management server 4 (step SF3), it generates a screen related to the vegetable compartment 15 (step SF4).

Step SF4 will now be described in detail.
The application execution unit 301 generates a vegetable compartment home screen GM5 having a case image display area HA6 in which a lower case image GCG indicated by the lower case image data 4125 included in the drawer food management record HR is superimposed with an upper case image JCG indicated by the upper case image data 4123 included in the same record.
Furthermore, when the memo input presence/absence information 4119 indicates that a memo exists, the application executing unit 301 generates a vegetable compartment home screen GM5 that displays a memo input icon MIC in association with the corresponding case image CG.

Step SF4 will now be described in detail.
The application execution unit 301 generates a food image list screen GM6 on which the photographed food images FG indicated by the food image data 4116 are arranged for each case CA.

Returning to the explanation of flowchart FF, the application execution unit 301 displays a screen related to the vegetable compartment 15 in response to the touch operation (step SF5).

Next, we will explain the operation of the food management system 1000 when operations are performed on the screen related to the vegetable compartment 15.

When the memo save button B1 is touched, the application execution unit 301 transmits information to the food management server 4 to update the memo input presence/absence information 4119 and memo information 4130. This information includes information indicating whether the operation target is the upper case image JCG or the lower case image GCG, memo input presence/absence information 4119 indicating that a memo exists, and memo information 4130 indicating the memo that has been input. The food management server 4 appropriately updates the withdrawn food management record HR based on the received information.

When the memo erase button B2 is touched, the application execution unit 301 transmits information to the food management server 4 to update the memo input presence/absence information 4119 and the memo information 4130. This information includes information indicating whether the memo is to be erased from the upper case image JCG or the lower case image GCG, and the memo input presence/absence information 4119 indicating that no memo is present. The food management server 4 appropriately updates the withdrawn food management record HR based on the received information.

The vegetable compartment 15 is an example of a drawer-type storage compartment described using Figures 23-25. However, the screen displays and operations of the food management system 1000 shown in Figures 23-25 may also be performed in the same way for the freezer compartment 14. In this case, the database shown in Figure 10 is a database that stores various information related to the food in the freezer compartment 14.

[1-3. Effects]
As described above, camera unit 2 comprises main body 201 installed on top surface 10B of refrigerator 1, and imaging member 202 extending from main body 201 to the front of refrigerator 1 and equipped with camera 213 that captures images from the upper front to the lower front of refrigerator 1. Main body 201 comprises position restricting member 205 that abuts front edge 10A of refrigerator 1 and restricts the position of imaging member 202 in the front-to-rear direction of refrigerator 1.

As a result, the position of the imaging member 202 in the front-to-rear direction of the refrigerator 1 is restricted by the position restricting member 205, so that the camera 213 is positioned so that it can capture images from the upper front to the lower front of the refrigerator 1. As a result, food entering and leaving the refrigerator 1 can be accurately included in the images captured by the camera 213, and food entering and leaving the refrigerator 1 can be accurately photographed.

The position restriction member 205 is a position restriction piece that extends downward from the bottom surface 206 of the main body 201 when installed.

This allows the position of the camera unit 2 to be regulated by utilizing the gap formed between the main box body 10 and the door 11C. Therefore, the camera unit 2 can be installed in an existing refrigerator 1, so that food entering and leaving the refrigerator 1 can be photographed with high accuracy even in an existing refrigerator 1.

The position control member 205 is formed to be shorter than the length to the packing provided between the refrigerator 1 and the door 11C of the refrigerator 1.

This allows the position of the camera unit 2 to be regulated by utilizing the gap formed between the main box body 10 and the door 11C without impeding the cooling effect of the refrigerator compartment 11. Therefore, food being put in and taken out of the refrigerator 1 can be photographed with high accuracy without impeding the cooling effect of the refrigerator compartment 11.

The position control member 205 extends in the width direction of the main body 201.

This allows the camera unit 2 to have a large contact area with the main box 10, stabilizing the installation position of the camera unit 2. This allows food being taken in and out of the refrigerator 1 to be photographed with greater accuracy.

The position control member 205 has a mark 207 for adjusting the position of the camera unit 2 in the width direction of the refrigerator 1.

By using the mark 207 as a guide, the user P can easily install the camera unit 2 in the left-right direction of the refrigerator 1. Therefore, the user P can easily install the camera unit 2 so that the food being taken in and out of the refrigerator 1 can be photographed with high accuracy.

The position control member 205 has a mark 207 at the center of the width of the main body 201.

By using the mark 207 as a landmark, the user P can easily install the camera unit 2 in the center of the refrigerator 1 in the left-right direction. Therefore, the user P can easily install the camera unit 2 so that the food being taken in and out of the refrigerator 1 can be photographed with greater accuracy.

As described above, the camera unit 2 includes a camera 213 that captures moving images of the refrigerator 1 from above the front of the refrigerator 1, an imaging control unit 223 that causes the camera 213 to start capturing images when the human presence sensor 212 that detects whether or not there is a person around the refrigerator 1 detects that there is a person around the refrigerator 1, and a recording control unit 224 that starts recording the images captured by the camera 213 at a predetermined timing after the imaging control unit 223 has caused the camera 213 to start capturing images.
In addition, the control method of camera unit 2 is such that, when the human presence sensor 212, which detects whether or not there is a person around the refrigerator 1, detects that there is a person around the refrigerator 1, it causes the camera 213 to start capturing images, and at a predetermined timing after the camera 213 has started capturing images, it starts recording the images captured by the camera 213.

According to the camera unit 2 and the control method for the camera unit 2, when the human presence sensor 212 detects the presence of a person near the refrigerator 1, the camera 213 is caused to start capturing images, thereby increasing the likelihood that food entering or leaving the refrigerator 1 before the door 11C or drawer 16A of the refrigerator 1 is opened will be included in the captured images of the camera 213. This allows food entering or leaving the refrigerator 1 to be captured with high accuracy.

After the photography control unit 223 causes the camera 213 to start photographing, when the door 11C or the drawer 16A of the refrigerator 1 is opened, the recording control unit 224 starts recording the photographed results of the camera 213.

By doing this, recording starts at a timing when there is a high possibility that food will start being taken in or out of the refrigerator 1, which prevents unnecessary recording in a situation where food is not being taken in or out of the refrigerator 1. In addition, the recording control unit 224 can prevent the data capacity of the recording file 231 from increasing, which prevents an increase in the processing target of the food detection unit 225 and an increase in the processing time and processing load of the food detection unit 225.

If the door 11C or drawer 16A of the refrigerator 1 is closed after recording has started, the recording control unit 224 stops recording.

By doing this, recording ends when there is a high possibility that food has been taken in or taken out of the refrigerator 1, which prevents unnecessary recording in a situation where no food is being taken in or out of the refrigerator 1. In addition, the recording control unit 224 can prevent the data capacity of the recording file 231 from increasing, which prevents an increase in the processing target of the food detection unit 225 and an increase in the processing time and processing load of the food detection unit 225.

When the recording control unit 224 stops recording, it generates a single recording file 231 containing the captured images from when the door 11C or drawer 16A of the refrigerator 1 is opened to when it is closed.

In this way, the image capturing results from when the door 11C or drawer 16A of the refrigerator 1 is opened to when it is closed are generated as one recording file 231, so that the one recording file 231 can be generated quickly while preventing the data capacity of the one recording file 231 from increasing. In addition, since the one recording file 231 can be generated quickly while preventing the data capacity of the one recording file 231 from increasing, the food detection unit 225 can quickly start processing related to the recording file 231 after food is stored in or removed from the refrigerator 1.

If the human presence sensor 212 detects that no human is present around the refrigerator 1 after the recording control unit 224 has stopped recording, the imaging control unit 223 causes the camera 213 to end imaging.

According to this, when the human presence sensor 212 detects that no human is present around the refrigerator 1, the camera 213 is made to stop capturing images, thereby preventing the camera 213 from capturing images when no food is being taken in or out of the refrigerator 1. This makes it possible to accurately capture images of food being taken in and out of the refrigerator 1 while preventing an increase in power consumption.

The camera unit 2 is equipped with a food detection unit 225 that detects food entering or leaving the refrigerator 1 based on the recording results of the recording control unit 224.

This allows food to be detected based on the results of photographing the food being taken in and out of refrigerator 1 with high accuracy, so food being taken in and out of refrigerator 1 can be detected with high accuracy.

As explained above, the food management app 311 causes the terminal processor 300 of the terminal device 3 equipped with a touch panel 32 to function as an app execution unit 301 that displays an image of the interior of the refrigerator compartment RG using the touch panel 32 and displays a photographed food image FG in a divided area BA in the displayed image of the interior of the refrigerator compartment RG that corresponds to the storage area of the food stored in the refrigerator compartment 11.
The terminal device 3 also includes a touch panel 32 and an app execution unit 301 that displays an image of the interior of the refrigerator compartment RG using the touch panel 32 and displays a photographed food image FG in a divided area BA in the displayed image of the interior of the refrigerator compartment RG that corresponds to the storage area of the food stored in the refrigerator compartment 11.
In addition, the control method of the terminal device 3 displays a refrigerator compartment interior image RG, and in the displayed refrigerator compartment interior image RG, displays a photographed food image FG in a divided area BA corresponding to the storage area of the food stored in the refrigerator compartment 11.

According to the food management app 311, the terminal device 3, and the control method of the terminal device 3, the photographed food image FG is displayed in a divided area BA that corresponds to the storage area of the food stored in the refrigerator in an image showing the interior of the refrigerator compartment. This allows the user P to intuitively grasp what kind of food is stored in the refrigerator compartment 11.

The photographed food image FG is an image captured by the refrigerator compartment camera 203.

By doing this, user P is provided with photographed images of the food that is actually stored in refrigerator 1, allowing user P to more intuitively grasp what kind of food is stored in refrigerator compartment 11.

The refrigerator compartment interior image RG is an image that corresponds to the model of refrigerator 1 used by user P of the terminal device 3.

By doing this, the user P is provided with a refrigerator compartment interior image RG that corresponds to the refrigerator compartment 11 of the refrigerator 1 that is actually being used, allowing the user P to more intuitively grasp what kind of food is stored in the refrigerator compartment 11.

The refrigerator compartment interior image RG is divided into multiple divided areas BA. The number of divided areas BA that the refrigerator compartment interior image RG has depends on the model of the refrigerator 1 used by the user P of the terminal device 3.

This allows user P to understand what foods are stored in the storage area of refrigerator compartment 11 of refrigerator 1 that is actually being used. Therefore, user P can more intuitively understand what foods are stored in refrigerator compartment 11.

When displaying multiple photographed food images FG in one divided area BA, the application execution unit 301 displays the multiple photographed food images FG side by side.

This allows multiple photographed food images FG to be displayed clearly for each divided area BA, allowing the user P to intuitively and easily grasp what kind of food is stored in the refrigerator compartment 11.

A maximum number of photographed food images FG that can be displayed simultaneously is set for each divided area BA.

This prevents a decrease in visibility when there are many photographed food images FG displayed in one divided area BA. Therefore, the user P can intuitively and easily grasp what kind of food is stored in the refrigerator compartment 11.

When the number of photographed food images FG to be displayed in one divided area BA exceeds the upper limit, the application execution unit 301 switches between and displays the photographed food images FG within the range of the upper limit.

This prevents a decrease in visibility when there are many photographed food images FG displayed in one divided area BA, while allowing the user P to view the photographed food images FG that cannot be displayed.

The application execution unit 301 accepts memo input from the user P of the terminal device 3 for each photographed food image FG, and displays a memo input icon MIC indicating that a memo has been accepted in association with the photographed food image FG.

This allows user P to know which food the note has been entered for. Therefore, user P can intuitively know what kind of food is stored in the refrigerator compartment 11, and can also know which food the note has been entered for.

The food management app 311 is an application program that can be installed on the terminal device 3.

As a result, a terminal device 3 that does not have the function of displaying the refrigerator compartment interior image RG and the photographed food image FG can be turned into a terminal device 3 that can display these images by installing the food management app 311.

As described above, the food management app 311 causes the terminal processor 300 of the terminal device 3 equipped with the touch panel 32 to function as an app execution unit 301 that displays images captured by the drawer camera 204 capturing images of the drawer 16A of the refrigerator 1 having an upper case JCA and a lower case GCA. The app execution unit 301 is capable of displaying the upper case image JCG and the lower case image GCG, and accepts memo input from the user P of the terminal device 3 for each of the upper case image JCG and the lower case image GCG. When the app execution unit 301 accepts memo input from the user P for the upper case image JCG, it displays a memo input icon MIC indicating that memo input has been accepted in association with the upper case image JCG, and when it accepts memo input from the user P for the lower case image GCG, it displays the memo input icon MIC in association with the lower case image GCG.

The terminal device 3 also includes a touch panel 32 and an application execution unit 301 that displays an image captured by the drawer camera 204. The application execution unit 301 is capable of displaying an upper case image JCG and a lower case image GCG, and accepts memo input from a user P of the terminal device 3 for each of the upper case image JCG and the lower case image GCG. When the application execution unit 301 accepts memo input from the user P for the upper case image JCG, it displays a memo input icon MIC indicating that memo input has been accepted in association with the upper case image JCG, and when the application execution unit 301 accepts memo input from the user P for the lower case image GCG, it displays the memo input icon MIC in association with the lower case image GCG.

The control method of the terminal device 3 is also capable of displaying an upper case image JCG and a lower case image GCG, and accepts memo input from a user P of the terminal device 3 for each of the upper case image JCG and the lower case image GCG, and when memo input from the user P is accepted for the upper case image JCG, a memo input icon MIC indicating that memo input has been accepted is displayed in association with the upper case image JCG, and when memo input from the user P is accepted for the lower case image GCG, the memo input icon MIC is displayed in association with the lower case image GCG.

According to this, the food management app 311, the terminal device 3, and the control method of the terminal device 3 display a photographed image of the case CA of the drawer 16A, and the user P can input a note into this displayed photographed image. This allows the user P to intuitively grasp the food stored in the drawer 16A of the refrigerator 1, and improves the convenience of the user P.

The application execution unit 301 displays the upper case image JCG superimposed on the lower case image GCG, and when an operation to move the upper case image JCG is received from the user P, the application execution unit 301 moves the first case image and displays the lower case image GCG.

This allows the lower case image GCG to be displayed in a manner that moves the actual case CA. This allows the user P to more intuitively grasp the foods stored in the drawer 16A of the refrigerator 1, and improves the convenience for the user P.

When the application execution unit 301 displays a memo input icon MIC in association with the upper case image JCG, it moves the memo input icon MIC together with the upper case image JCG.

This makes it possible to maintain the association between the upper case image JCG and the memo input icon MIC even when the user P moves the upper case image JCG. Therefore, even when the user P moves the upper case image JCG, the user P can properly understand which image the memo input icon MIC refers to, improving the convenience for the user P.

The application execution unit 301 displays a switch button for switching the captured image to be displayed between the upper case image JCG and the lower case image GCG, and switches the captured image to be displayed between the upper case image JCG and the lower case image GCG in response to an operation on the switch button.

This allows the user P to easily switch the case image CG displayed by operating the switch button, improving convenience for the user P.

The upper case image JCG is an image captured when the upper case JCA is pulled out from the refrigerator 1 by more than a predetermined threshold. The lower case image GCG is an image captured when the lower case GCA is pulled out from the refrigerator 1 by more than a predetermined threshold.

This allows the captured image that shows as much food as possible to be used as the case image CG, allowing the user P to intuitively grasp the food stored in the drawer 16A of the refrigerator 1 and improving the convenience for the user P.

The food management app 311 is an application program that can be installed on the terminal device 3.

As a result, a terminal device 3 that does not have the function of displaying a case image CG or the function of accepting input of notes for the displayed case image CG can be made into a terminal device 3 that has these functions by installing the food management application 311.

(Embodiment 2)
Next, a description will be given of embodiment 2. In the description of embodiment 2, the same components as those of each part of food control system 1000 of embodiment 1 will be given the same reference numerals and detailed description will be omitted as appropriate.

[2-1. Configuration]
FIG. 26 is a block diagram showing the configuration of the terminal device 3 and the food management server 4 in the second embodiment.
As is clear from a comparison between Figures 26 and 7, the server processor 400 in embodiment 2 functions as a server communication control unit 401, an information processing unit 402, and a food recognition unit 403 by executing a control program stored in the server memory unit 410.

The food recognition unit 403 performs food recognition processing based on the photographed food image FG indicated by the food image data 4116 included in each of the first update information and the second update information.

The food recognition process is a process of recognizing food from the photographed food image FG and retrieving an icon representing the same type of food as the recognized food from the icon database. This icon database stores an icon representing each food. Hereinafter, this icon will be referred to as a food icon. The icon database may be stored in the food management server 4 or in an external device with which the food management server 4 can communicate.
In the present embodiment, the food icon corresponds to an example of a food image of the present disclosure.

For example, the food recognition unit 403 recognizes food as follows.
The food recognition unit 403 identifies an image that is most similar to the photographed food image FG indicated by the food image data 4116 included in the update information from among the images stored in the food image database. The food image database is a database in which image data for each food is stored. The degree of similarity is determined based on features such as shape and color. The food recognition unit 403 then recognizes that the food corresponding to the identified image is the food indicated by the food image data 4116 included in the update information.

The food recognition unit 403 may recognize food using AI (Artificial Intelligence). For example, a machine learning process is performed to learn feature quantities such as the color and shape of food from the photographed food image FG for training, and a trained model is constructed. This trained model is stored in the server storage unit 410. The food recognition unit 403 uses the food image data 4116 included in the update information as input data, refers to the trained model, and recognizes the food indicated by the photographed food image FG. Here, the trained model may be stored in a device other than the food management server 4 connected to the global network GN, and the food may be recognized in the food management server 4. In this configuration, the food recognition unit 403 transmits the food image data 4116 included in the update information to a device that stores the trained model, and acquires the food identification result from this device. The food recognition unit 403 then recognizes the food indicated by the acquired identification result as the food indicated by the photographed food image FG indicated by the food image data 4116 included in the update information.

When the food recognition unit 403 recognizes a food, it obtains a food icon for the recognized food from the icon database. Then, in the second embodiment, the icon data of the food icon obtained by the food recognition unit 403 is appropriately stored instead of or in addition to the food image data 4116 held in the refrigerator food management record RR. Also, in the second embodiment, the icon data of the food icon obtained by the food recognition unit 403 is appropriately stored instead of or in addition to the food image data 4116 held in the drawer food management record HR.

As a result, in embodiment 2, a food icon can be displayed in place of the photographed food image FG in the app UI 320.

[2-2. Effects, etc.]
According to the second embodiment, the same effects as those of the first embodiment are achieved.

Other Embodiments
As described above, the above-mentioned embodiment has been described as an example disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can be applied to an embodiment in which modifications, substitutions, additions, omissions, etc. are made. In addition, it is also possible to combine the components described in the above-mentioned embodiment to create a new embodiment.
Therefore, other embodiments will be exemplified below.

In the above-described embodiment, a refrigerator 1 is used as an example of a storage facility. However, the storage facility of the present disclosure is not limited to a refrigerator 1, and may be, for example, a warehouse installed in an office or the like. In the above-described embodiment, food is used as an example of an item of the present disclosure, but the item of the present disclosure is not limited to food, and may be other items such as books depending on the storage facility.

In the above-described embodiment, a configuration has been exemplified in which the position restriction member 205 has the mark 207 at the center in the left-right direction of the main body 201. However, the position of the mark 207 is not limited to the center in the left-right direction of the main body 201. The position of the mark 207 may be other than the center in the left-right direction of the main body 201. In this case, each mark is appropriately recorded on the central position recording paper CY so that the camera unit 2, whose position is adjusted based on the mark 207, can appropriately perform various detections.

In the above-described embodiment, the refrigerator compartment interior image RG, which is an image of the refrigerator compartment 11 viewed from the front when the left door 11A and the right door 11B are open, is exemplified as the refrigerator compartment interior image of the present disclosure. However, the refrigerator compartment interior image of the present disclosure is not limited to this refrigerator compartment interior image RG, and may be an image of the refrigerator compartment viewed from the front when a single door is open, or an image of the interior of a refrigerator 1 that includes a refrigerator compartment and a freezer compartment and is viewed from the front when the doors are open.

The shooting range of the drawer camera 204 may be a range that combines one or more drawer-type storage rooms depending on the type of drawer-type storage room on the above screen displayed by the app UI 320.

In addition to the storage date and time, the food management system 1000 may also manage other elements such as remaining quantity and expiration date.

The main body 201 and the photographing member 202 of the camera unit 2 may be configured separately.

The camera unit 2 may be configured to transmit the case image CG to the food management server 4 even if the food is not pulled out by the predetermined threshold or more. In this configuration, the food management server 4 stores the case image CG when the food is not pulled out by the predetermined threshold or more, and provides it to the terminal device 3. The application execution unit 301 then displays this case image CG in the application UI 320 so that it can be replaced with each of the upper case image JCG and the lower case image GCG.

In the above embodiment, the case where the user P installs the camera unit 2 in the refrigerator 1 is illustrated, but the camera unit 2 may be shipped in a state where it is already installed in the refrigerator 1.

The server processor 400 may function as the food detection unit 225, and the camera unit processor 220 may not function as the food detection unit 225. In this configuration, the camera unit 2 transmits the recording file 231 to the food management server 4.

In the above-described embodiment, the human presence sensor 212 is exemplified as a sensor of the present disclosure. However, the sensor of the present disclosure is not limited to the human presence sensor 212, and may be an illuminance sensor, an electrostatic sensor provided on the grip of the refrigerator 1, or the like. In the case of this electrostatic sensor, the refrigerator 1 and the camera unit 2 are capable of communicating with each other, and the refrigerator 1 transmits the detection value of the electrostatic sensor to the camera unit 2.

The refrigerator food management record RR and the drawer food management record HR may be stored in the terminal device 3, and the food management app 311 may manage various information related to food detected by the camera unit 2. In this configuration, the app execution unit 301 appropriately updates the contents of each record stored in the terminal device 3 in the same manner as the food management server 4 described above. This simplifies the system configuration, as the food management system 1000 does not need to include the food management server 4. Furthermore, in this configuration, the app execution unit 301 may function as the food recognition unit 403.

For example, the types of compartments formed in the main box 10 of the refrigerator 1 are not limited to the refrigerator compartment 11, ice-making compartment 12, fresh-freezing compartment 13, freezer compartment 14, and vegetable compartment 15, and there may be fewer compartments, or other types of compartments may be formed. Also, the number of doors provided at the opening on the front of the refrigerator compartment 11 may be one.

For example, the number of cases provided in drawers 14A and 15A is not limited to two, and may be greater. In this case, one case corresponds to the first case of the present disclosure, and the case below this case corresponds to the second case of the present disclosure.

For example, the camera unit processor 220, the terminal processor 300, and the server processor 400 may be configured as a single processor or as multiple processors.

The components shown in Figures 6, 7, and 26 are merely examples, and the specific implementation form is not particularly limited. In other words, it is not necessary to implement hardware that corresponds to each component individually, and it is of course possible to implement a configuration in which a single processor executes a program to realize the functions of each component. Also, some of the functions realized by software in the above-mentioned embodiment may be hardware, or some of the functions realized by hardware may be software. In addition, the specific detailed configurations of the other components of the camera unit 2, terminal device 3, and food management server 4 may also be changed as desired without departing from the spirit of this disclosure.

In addition, for example, the step units of the operations shown in Figures 13, 14, 22, and 25 are divided according to the main processing content in order to facilitate understanding of the operation of each part of the food management system 1000, and the present disclosure is not limited by the manner in which the processing units are divided or the names of the processing units.

The above-described embodiments are intended to illustrate the technology disclosed herein, and various modifications, substitutions, additions, omissions, etc. may be made within the scope of the claims or their equivalents.

As described above, the program, terminal device, and control method for the terminal device according to the present invention can be used to display images of food being put into and taken out of a refrigerator.

1. Refrigerator (storage)
2. Camera unit (photography device)
3 Terminal device 4 Food management server 10 Main box body 10A Front edge 10B Top surface 11 Refrigerator compartment 11A Left door 11B Right door 11C Door 12 Ice making compartment (drawer type storage compartment)
12A, 13A, 14A, 15A, 16A Drawers 13 Fresh freezing chamber (drawer-type storage chamber)
14 Freezer (drawer-type storage compartment)
15 Vegetable compartment (drawer-type storage compartment)
32 Touch panel (display unit)
201 Main body 202 Shooting member 203 Refrigerator camera 204 Drawer camera 205 Position control member 206 Bottom surface 207 Mark 212 Human sensor (sensor)
213 Camera 223 Photography control unit 224 Recording control unit 225 Food detection unit (item detection unit)
231 Recording file 300 Terminal processor (computer)
301 Application execution unit (display control unit)
311 Food management application (program, application program)
1000 Food Control System BA Division Area (Area)
FG, FG1 Photographed food images (food images)
GCA Lower Case (2nd Case)
GCG lower case image (second case image)
JCA Upper Case (1st Case)
JCG upper case image (first case image)
MIC memo input icon (memo input image)
P User RG Refrigerator interior image (interior image)

Claims (8)

A computer of a terminal device having a display unit,
The display unit functions as a display control unit that displays an interior image showing the interior of an empty refrigerator as viewed from the front of the refrigerator, and displays a food image showing the food in an area of the displayed interior image corresponding to an area for storing the food in the refrigerator,
The interior image is an image corresponding to the model of the refrigerator used by the user of the terminal device,
The interior image is divided into a plurality of areas, and the number of areas included in the interior image corresponds to the model of the refrigerator used by a user of the terminal device,
The display control unit is
displaying the food images showing the foods detected to have been stored in the storage compartment, excluding the foods detected to have been removed from the storage compartment;
When a plurality of the food images are displayed in one of the areas, the plurality of the food images are displayed side by side;
An upper limit of the number of food images that can be displayed simultaneously is set in one of the areas,
The display control unit is
When the number of the food images to be displayed in one of the areas corresponding to the food storage area exceeds the upper limit number, the food images to be displayed are displayed in the area in a number equal to or less than the upper limit number;
switching the food images displayed in the area while maintaining the number of the food images to be displayed in the area equal to or less than the upper limit number in response to an operation on the food images displayed in the area;
program. The display control unit displays the food image indicating the food detected to have been stored in the storage unit in an area corresponding to the storage area in which the food has been stored.
The program according to claim 1. The food image is an image captured by a refrigerator compartment camera that captures food entering and leaving the refrigerator.
The program according to claim 1. The food image is an icon corresponding to the type of food.
The program according to claim 1. The display control unit is
accepting an input of a memo from a user of the terminal device for each of the food images, and displaying a memo input image indicating that a memo has been accepted in association with the food image;
The program according to any one of claims 1 to 4. An application program that can be installed in the terminal device,
The program according to any one of claims 1 to 5. A display unit;
A display control unit that displays an interior image showing the interior of an empty refrigerator as viewed from the front of the refrigerator by the display unit, and displays a food image showing the food in an area corresponding to a storage area of the food stored in the refrigerator in the displayed interior image,
The interior image is an image corresponding to the model of the refrigerator used by a user of the terminal device,
The interior image is divided into a plurality of areas, and the number of areas included in the interior image corresponds to the model of the refrigerator used by a user of the terminal device,
The display control unit is
displaying the food images showing the foods detected to have been stored in the storage compartment, excluding the foods detected to have been removed from the storage compartment;
When a plurality of the food images are displayed in one of the areas, the plurality of the food images are displayed side by side;
An upper limit of the number of food images that can be displayed simultaneously is set in one of the areas,
The display control unit is
When the number of the food images to be displayed in one of the areas corresponding to the food storage area exceeds the upper limit number, the food images to be displayed are displayed in the area in a number equal to or less than the upper limit number;
switching the food images displayed in the area while maintaining the number of the food images to be displayed in the area equal to or less than the upper limit number in response to an operation on the food images displayed in the area;
Terminal device. A method for controlling a terminal device having a display unit, comprising:
displaying an interior image showing the interior of an empty refrigerator as viewed from the front of the refrigerator by the display unit;
The interior image is an image corresponding to the model of the refrigerator used by the user of the terminal device,
The interior image is divided into a plurality of areas , and the number of areas included in the interior image corresponds to the model of the refrigerator used by the user of the terminal device,
In the displayed interior image, food images showing the foods detected to have entered the interior of the refrigerator, excluding the foods detected to have been removed from the interior of the refrigerator, are displayed in the area corresponding to the storage area of the foods stored in the refrigerator;
When a plurality of the food images are displayed in one of the areas, the plurality of the food images are displayed side by side;
An upper limit of the number of food images that can be displayed simultaneously is set in one of the areas,
When the number of the food images to be displayed in one of the areas corresponding to the food storage area exceeds the upper limit number, the food images to be displayed are displayed in the area in a number equal to or less than the upper limit number;
switching the food images displayed in the area while maintaining the number of the food images to be displayed in the area equal to or less than the upper limit number in response to an operation on the food images displayed in the area;
A method for controlling a terminal device.

Images