JP7675908B2 - Imaging system and imaging device - Google Patents

Description

The present invention relates to an imaging system and an imaging device.

A refrigerator has been proposed that uses a camera to capture images of the interior of the refrigerator for the purpose of inventory management inside the refrigerator. According to the refrigerator described in Patent Document 1, the refrigerator "includes a plurality of storage compartments divided into at least one capture range, a plurality of interior cameras that capture at least one of the capture ranges, or at least one interior camera that captures the plurality of capture ranges, and a control device that stores images of the plurality of capture ranges in a storage device and communicates with an operation display device that displays the images." It is also described that, in the operation display device, "when an operation is performed to instruct the movement of the image inside the refrigerator compartment by touching and moving a finger left and right (a so-called swipe operation or flick operation performed on the touch panel of a smartphone or tablet terminal), the control device displays an image of the door pocket side of the door that is located in the opposite direction to the refrigerator compartment when the user observes the interior of the refrigerator compartment from the front side of the refrigerator, together with the image inside the refrigerator compartment."

Patent No. 6296908

The refrigerator described in Patent Document 1 requires the user to perform swipe or flick operations on the operation display device in order to simultaneously view the interior of the refrigerator and the door pocket.

The imaging system of the present invention, which has been made in consideration of the above circumstances, comprises a refrigerator having a storage compartment with an opening at the front and a revolving door that opens and closes the opening and has a door storage section, and an imaging device that is arranged outside the storage compartment and has a lens and an illumination section located forward of the edge surface of the opening, and is characterized in that a light-shielding wall that blocks direct light is provided between the illumination section and the lens, the imaging device is arranged so that the door storage section can be included in a part of the periphery of the imaging range, and the illumination section is arranged on the door storage section side of the lens. Other aspects of the present invention will be described in the embodiments described later.

FIG. 1 is a front view showing a refrigerator according to a first embodiment. FIG. 2 is a side view showing the refrigerator according to the first embodiment. FIG. 2 is a side view showing a case in which the imaging device according to the first embodiment is stored. 1 is a front view showing a state in which the door of a refrigerator according to a first embodiment is open. FIG. FIG. 2 is a top view showing the refrigerator according to the first embodiment with the door open. 1 is a front view showing an imaging device according to a first embodiment. 1 is a side view showing an imaging device according to a first embodiment. 1 is a side cross-sectional view showing an imaging device according to a first embodiment. 1 is a perspective view showing an imaging device according to a first embodiment. FIG. 1 is a configuration diagram showing an imaging system for a refrigerator according to a first embodiment. 5 is a flowchart showing a process at the time of manual shooting setting according to the first embodiment. 5 is a flowchart showing a process during automatic shooting setting according to the first embodiment. 4 is a flowchart showing image processing according to the first embodiment. 13 is an image showing the interior of the fridge before image processing. 11 is an image showing the cutting process of each storage section. 11 is an image showing the distortion correction process for each storage section. 13 is an image showing a rotation process of each storage section. 11 is an image showing keystone correction processing for each image. 11 is an image showing a process of combining images. A side view showing the state in which the drawer part of the refrigerator in the second embodiment is pulled out. 13 is an image showing a cutout process of a drawer section of a refrigerator according to a second embodiment. 13 is an image showing an enlargement process of a drawer section of a refrigerator according to a second embodiment. 13 is an image showing a synthesis process of each image including a pull-out portion according to the second embodiment. 13 is another example of an image showing a synthesis process of each image including a drawing portion according to the second embodiment. FIG. 2 is an explanatory diagram showing a normal state of the imaging device. FIG. 11 is an explanatory diagram showing a state in which the imaging device is in the middle of being stored. FIG. 2 is an explanatory diagram showing a state in which the imaging device is stored.

The following describes in detail the mode (embodiment) for carrying out the present invention with reference to the drawings as appropriate. The order of the control steps of the operation flows and the like disclosed in the embodiment can be interchanged or executed simultaneously as long as there is no mutual contradiction.
<<First embodiment>>
FIG. 1 is a front view showing the refrigerator 100 according to the first embodiment. FIG. 2A is a side view showing the refrigerator 100 according to the first embodiment. FIG. 2B is a side view showing the refrigerator 100 when the imaging device 30 according to the first embodiment is stored. FIG. 3 is a front view showing the refrigerator 100 according to the first embodiment with the door open. FIG. 4 is a top view showing the refrigerator 100 according to the first embodiment with the door open. In the following description, a six-door refrigerator 100 will be taken as an example, but is not limited to six doors.

As shown in FIG. 1, the insulated box (box 10) of the refrigerator 100 has storage compartments arranged in the following order from the top: refrigerator compartment 1, ice-making compartment 2, upper freezer compartment 3, vegetable compartment 4, and lower freezer compartment 5, which are arranged on the left and right. The refrigerator 100 has doors that open and close the openings of each storage compartment. These doors are rotating refrigerator compartment doors 1a and 1b divided into left and right, which open and close the opening of the refrigerator compartment 1, and pull-out ice-making compartment door 2a, upper freezer compartment door 3a, vegetable compartment door 4a, and lower freezer compartment door 5a, which open and close the openings of the ice-making compartment 2, upper freezer compartment 3, vegetable compartment 4, and lower freezer compartment 5, respectively. The positions of the vegetable compartment 4 and lower freezer compartment 5 may be changed in the vertical arrangement. The ice-making compartment 2, upper freezer compartment 3, and lower freezer compartment 5 are collectively referred to as the freezer compartment 7.

To secure the refrigerator compartment doors 1a and 1b to the refrigerator 100, door hinges (reference numerals omitted) are provided at the top and bottom of the refrigerator compartment 1. As shown in Figs. 3 and 4, when the refrigerator compartment doors 1a and 1b are opened and closed, door storage sections 11a (door storage section a) and 11b (door storage section b) are provided.

Refrigerator compartment 1 is a refrigerated storage compartment whose interior is kept in the refrigeration temperature range (0°C or higher), for example at an average of about 4°C. Freezer compartment 7 is a refrigerated storage compartment whose interior is kept in the freezing temperature range (less than 0°C), for example at an average of about -18°C. Vegetable compartment 4 is a refrigerated storage compartment whose interior is kept in the refrigeration temperature range, for example at an average of about 6°C, which prevents food from drying out.

In the refrigerator 100, the imaging device 30 for photographing the interior is located approximately in the center of the upper outer part of the storage compartment. That is, it is provided approximately in the center above the box body (insulated box body) of the refrigerator 100. When the box door is closed, the lens 34 (see Figures 5A and 5B) of the imaging device 30 is located forward of the opening edge surface 10a of the storage compartment, and preferably forward of the refrigerator compartment doors 1a, 1b (doors). That is, the imaging device 30 is provided outside the storage compartment, and can capture an overview of the situation inside the refrigerator, including the door storage sections 11a, 11b.

The imaging device 30 also has a mechanism for sliding back and forth (a mechanism for sliding in a direction perpendicular to the storage compartment opening), and can move back and forth across the front door of the refrigerator 100. For example, as shown in FIG. 2B, it can be moved behind the front door or behind the edge of the storage compartment opening, so it does not affect the depth dimension when packed. Furthermore, the number of refrigerators that can be loaded onto a transport truck is not reduced, and even after installation, the imaging device 30 can be stored and the imaging range concealed when not in use. By concealing it, the configuration can be designed with consideration for users who do not like the camera's imaging area to always be within their living area.

The refrigerator door 1b is provided with a shooting button 39 (a shooting command receiving unit that receives shooting commands from the user) that is used when manually taking a picture with the imaging device 30. The shooting button 39 may be provided on the refrigerator door 1a or in another area of the box or door.

Fig. 5A is a front view showing the imaging device 30 according to the first embodiment. Fig. 5B is a side view showing the imaging device 30 according to the first embodiment. Fig. 5C is a side cross-sectional view (cross-sectional view AA in Fig. 5A) showing the imaging device 30 according to the first embodiment. Fig. 6 is a perspective view showing the imaging device 30 according to the first embodiment.

The imaging device 30 is composed of a case 31 and a cover 32. In front of the case 31, there is a camera 33 (imaging unit) and a fisheye lens 34. As shown in FIG. 6, the lens 34 is surrounded by a light-shielding wall 35. There are multiple lighting units 38 near the light-shielding wall 35.

As shown in FIG. 5C, the case 31 contains a control SoC 41 and a wireless LAN unit 42. The control SoC 41 (system on a chip) is an integrated circuit that integrates functions such as a processor core and other functions found in general microcontrollers, as well as functions for application purposes, on a single chip of the integrated circuit, and is designed to function in cooperation as a system. Note that the control SoC 41 is an example of an integrated circuit, and may be replaced by other integrated circuits, etc.

An imaging device 30 installed outside the storage compartment captures bird's-eye images of the door storage sections 11a and 11b and the interior of the storage compartment.

The features of the imaging device 30 will be further described.
(1) The camera 33 used in the imaging device 30 can capture images at a wide angle of 180 degrees or more, and has a fisheye lens, so that the entire interior of the fridge can be viewed from above when the door is opened.
(2) The camera lens 34 of the imaging device 30 is positioned in front of the front surface of the refrigerator door, so that the condition inside the refrigerator can be photographed even when only the vegetable compartment 4 and the drawer section 6 of the lower freezer compartment 5 are open.

(3) The imaging device 30 is equipped with an illumination unit 38, and by capturing images using the exterior illumination unit 38 that can be adjusted to an illumination level suitable for capturing images, images with high visibility can be obtained. The illumination provided inside the refrigerator is relatively high in illumination, so when captured by a camera, the stored items are strongly reflected, and the captured images tend to be overexposed. For this reason, in this embodiment, when capturing images, the interior illumination is turned off and the exterior illumination unit 38 is turned on. In addition, the illumination of the illumination unit 38 is directional so as to illuminate the interior of the refrigerator, the door storage units 11a and 11b when the door is open, and the entire second storage compartment (e.g., the vegetable compartment 4 and the lower freezer compartment 5).
(4) The imaging device 30 has the light-shielding wall 35, which prevents direct light from the lighting unit 38 from entering the shooting range, thereby making it possible to obtain an image with high visibility and reduced reflections and overexposure.
(5) By providing a light-shielding wall 35 between the lighting unit 38 of the imaging device 30 and the lens 34 to block direct light, the lighting can be installed as close as possible to the lens 34, resulting in a compact external shape and also providing the effect of preventing the lens 34 from being scratched.

(6) Imaging device 30 has a mechanism for sliding back and forth and can move back and forth across the front surface of the door of refrigerator 100, so that imaging device 30 does not affect the depth dimension during packaging. In addition, the number of refrigerators that can be loaded onto a transport truck is not reduced, and even after installation, imaging device 30 can be stored to hide the imaging range when not in use.
(7) The imaging device 30 includes a communication unit (wireless LAN unit 42), and image data transmitted from the communication unit is supplied to a server 54 (see FIG. 7) via, for example, a wireless router in the home (for example, a home router 53 (see FIG. 7)). Details will be described later.
(8) The imaging device 30 has a communication circuit that is connected to the refrigerator main body for power supply and image capture command transmission. The refrigerator main body receives power from a commercial power source or the like. Power for the imaging device 30 can be received from the commercial power source via the refrigerator main body. Note that the imaging device 30 may be driven by a rechargeable battery.

Figure 7 is a configuration diagram showing the imaging system of the refrigerator 100 according to the first embodiment. Door sensors (not shown) are also provided on the door side to detect the open/closed states of the refrigerator doors 1a and 1b, the ice maker door 2a, the upper freezer door 3a, the vegetable door 4a, and the lower freezer door 5a.

At the top of the refrigerator 100, a main microcomputer 52 (processor, control device, controller) is located, which is equipped with a CPU (Central Processing Unit), which is part of the control device, memories such as ROM (Read Only Memory) and RAM (Random Access Memory), and an interface circuit. The main microcomputer 52 controls the imaging system that captures images of the interior of the refrigerator based on information from the door sensor or the image capture button 39. The main microcomputer 52 also supplies power to the imaging device 30.

When the main microcomputer 52 is set to automatic shooting, based on information from the door sensor such as the door opening angle, if it detects an angle equal to or greater than a predetermined angle, it sends a shooting command to the control SoC 41. The control SoC 41 issues a shooting command to the camera 33 and acquires unprocessed image data from the camera 33. The control SoC 41 performs image processing on the unprocessed image data, for example, as shown in FIG. 10. On the other hand, when the main microcomputer 52 is set to manual shooting, if it detects that the shooting button 39 has been operated, it sends a shooting command to the control SoC 41. Subsequent processing is the same as when the main microcomputer 52 is set to automatic shooting.

As described above, the imaging device 30 is provided with a wireless LAN unit 42 that can be connected to the home router 53. By providing this wireless LAN unit 42, the interior image data processed by the control SoC 41 can be transmitted to the server 54 via the home router 53. The server 54 transmits the interior image data in response to a request to display an interior image from a mobile device such as a smartphone 55 or a personal computer owned by the user. This allows the user to grasp the status of the interior of the fridge.

Figure 8 is a flowchart showing the process when manual photography is set according to the first embodiment. Description will be made with reference to Figure 7 as appropriate. When the setting from the setting device (not shown) is for camera photography setting (camera setting ON) (S31), the main microcomputer 52 uses the door sensor to determine whether or not the door is open (Door Open?) (S32), and if the door is open (S32, Yes), proceeds to S33. On the other hand, if the door is not open (S32, No), the main microcomputer 52 returns to S32.

In S33, the main microcontroller 52 determines whether the shooting button 39 has been operated (pressed), and if the shooting button 39 has been pressed (S33, Yes), the process proceeds to S34, and if the shooting button 39 has not been pressed (S33, No), the process returns to S32.

In S34, the main microcomputer 52 commands the control SoC 41 to take a picture, and turns off the interior lighting and turns on the exterior lighting (lighting unit 38) (S35). The control SoC 41 commands the camera 33 to take a picture (S36). The main microcomputer 52 then turns on the interior lighting and turns off the exterior lighting (lighting unit 38) (S37).

The camera 33 transmits the unprocessed image data to the control SoC 41 (S38), and the control SoC 41 performs the interior image correction process (image processing) (S39). Details of the image processing will be described later with reference to FIG. 10.

The control SoC 41 performs image output processing (image update) on the processed image data via the wireless LAN unit 42 and the home router 53 to the server 54 (S40). When the image has been updated, the server 54 notifies the user's mobile device or the like (S41).
The above is the process when manual shooting is set.

Figure 9 is a flow chart showing the processing when automatic photography is set according to the first embodiment. Description will be made with reference to Figure 7 as appropriate. Note that description of the same processing as in Figure 8 will be omitted. When the setting from the setting device (not shown) is for automatic camera photography (automatic camera setting ON) (S31A), the main microcomputer 52 uses the door sensor to determine whether or not the door is open (door open?) (S32), and if the door is open (S32, Yes), proceeds to S33A. On the other hand, if the door is not open (S32, No), the main microcomputer 52 returns to S32.

In S33A, the main microcomputer 52 determines whether the door opening angle is equal to or greater than a predetermined angle (e.g., door angle 85 degrees or greater), and if the door opening angle is equal to or greater than the predetermined angle (S33A, Yes), the process proceeds to S34, and if the door opening angle is less than the predetermined angle (S33A, No), the process returns to S32. S34 to S41 are the same as in Fig. 13. In addition, in S33A, in the case of a double-door refrigerator compartment 1, if the opening angle of at least one of the refrigerator compartment doors 1a, 1b is equal to the predetermined angle (S33A, Yes), the process proceeds to S34.
The above is the process when automatic shooting is set.

The features of the imaging system of the refrigerator 100 will be further described.
(1A) The refrigerator has a detection unit (e.g., main microcontroller 52) that detects the opening angle of the refrigerator door, and can automatically take a photograph when the detection unit determines that the angle is equal to or greater than a predetermined angle (e.g., 85 degrees).
(2A) When photographing, the interior lighting is turned off and the lighting of the imaging device 30 is turned on. This not only adjusts the illuminance to an appropriate level for photographing, but also has the effect of informing the user when it is time to take a photograph.
(3A) The refrigerator 100 has a detection unit (for example, installed in the drawer section 6) that detects when the drawer distance reaches a certain distance or more. The detection unit detects the operation of the drawer and automatically takes a photograph, thereby enabling the imaging range of the vegetable compartment and freezer compartment to be kept roughly constant.

(4A) The refrigerator door has a button for photographing (e.g., photo button 39). When the user presses the button for photographing, the interior lighting is turned off and the lighting of the imaging device 30 is turned on, after which a photograph is taken. This makes it easy to recognize the timing at which the photograph was taken.
(5A) The captured image is subjected to image correction processing, and the distorted image captured by the wide-angle camera is corrected to provide an image that is close to the user's line of sight.
(6A) An application service is provided that allows a user to check the images via a mobile phone, etc., and a detection unit is provided on the door or the drawer to automatically determine which storage room the captured image belongs to.

As described above, the refrigerator 100 of this embodiment has a storage compartment (e.g., refrigerator compartment 1) with an opening at the front, a door (e.g., refrigerator compartment door 1a, 1b) with a rotatable storage section (e.g., door storage section 11a, 11b) provided at the front of the storage compartment, a door opening angle detection means for detecting the opening angle of the door, a lighting device that detects the opening of the door by the door opening detection means and illuminates the storage compartment, an imaging device (e.g., imaging device 30) provided at approximately the center of the upper outside of the storage compartment, the imaging device having a wide-angle fisheye lens (e.g., lens 34), a lighting device that turns on when shooting (e.g., lighting section 38), and a communication means (e.g., wireless LAN unit 42) that transmits the captured image data. The refrigerator 100 has a control device (e.g., main microcomputer 52) that issues a shooting command to the imaging device when a predetermined opening angle is detected by the door opening angle detection means.

As shown in FIG. 11, when the door is open, the image captured by the imaging device can capture not only the storage compartment but the entire door storage section in one captured image due to the action of the wide-angle fisheye lens. Furthermore, it is possible to capture the vegetable compartment (e.g., vegetable compartment 4) and freezer compartment (e.g., lower freezer compartment 5) of the drawer storage section in this capturing range, eliminating the need to provide multiple imaging devices for each storage compartment as in Patent Document 1, and eliminating the need for complex and expensive drive devices and drive controls to change the capturing range.

The images captured by the imaging device are sufficient to grasp the state of the items stored inside the refrigerator, but the image is curved due to the effect of the fisheye lens, and objects closer to the imaging device are more distorted. In addition, because the images are taken from above the refrigerator, they are positioned differently from the way users normally view the refrigerator from the front, so there is room for improvement in terms of visibility.

According to this embodiment, by performing the following image processing (S39), it is possible to provide an interior image that is close to the positional relationship of the front view that a user normally uses to view the refrigerator, by converting the distorted and curved image captured by the imaging device 30 having a fisheye lens with a wide angle of view.

Fig. 10 is a flow chart showing image processing (S39) according to the first embodiment. Images for each process will be described with reference to Figs. 11 to 16. Fig. 11 is an image showing the interior of the storage unit before image processing (storage unit c as inside the refrigerator compartment 1, door storage unit a as door storage unit 11a, and door storage unit b as door storage unit 11b). Fig. 12 is an image showing the cropping process for each storage unit. Fig. 13 is an image showing the distortion correction process for each storage unit. Fig. 14 is an image showing the rotation process for each storage unit. Fig. 15 is an image showing the keystone correction process for each image. Fig. 16 is an image showing the synthesis process for each image.

The main microcomputer 52 inputs the captured image (see FIG. 11) (S391) and performs a sectorial cutout process (cutout process) to encompass the range of each storage compartment (S392). Specifically, the main microcomputer 52 cuts out an image of the interior of the refrigerator compartment 1 and an image of each of the door storage compartments 11a and 11b, for example, cuts out the image into a sector shape, as shown in FIG. 12 (see FIG. 12).

Next, the main microcomputer 52 performs distortion correction processing (distortion correction process) on the images of each storage section that are curved due to the action of the fisheye lens (S393). Specifically, as shown in FIG. 13, the main microcomputer 52 performs distortion correction processing on the internal image of the refrigerator compartment 1 and the curved images of the door storage sections 11a and 11b. This distortion correction processing can use various known correction algorithms, but for example, distortion can be corrected by executing a calibration process in which parameters are set in advance (for example, before the refrigerator is shipped).

Next, the main microcomputer 52 performs an image rotation process to rotate the image of each storage section that has been subjected to the distortion correction process so that the image is viewed from the front, which is the state observed by the user (S394). Specifically, as shown in FIG. 14, the main microcomputer 52 positions the door storage section 11a on the left side and the door storage section 11b on the right side so that the image is viewed from the front, which is the state observed by the user.

Next, the main microcomputer 52 performs distortion correction (keystone correction processing) on the image that is distorted trapezoidally in the depth direction to convert it into a rectangle (S395). Specifically, as shown in FIG. 15, the main microcomputer 52 performs keystone correction processing on the image inside the refrigerator compartment 1 and the images of the door storage sections 11a and 11b. For example, by performing perspective projection transformation and its complementary processing such as the nearest neighbor method and the Lanczos method, the image of the storage section that is roughly trapezoidal or sector-shaped can be converted into a roughly rectangular shape.

Next, the main microcomputer 52 performs image synthesis processing on the images created in S395 to create a single image (S396). This image is displayed on the application of the smartphone 55 (see FIG. 7). Note that the embodiment shows only one example, and each process may be performed in any order.

The first embodiment of the method for processing image data in a refrigerator processes image data captured by an imaging device equipped with a wide-angle lens that includes multiple food storage areas in its imaging range, and is characterized by including and executing a cut-out process (S392) for cutting out multiple parts from the image data captured by the imaging device 30, a synthesis process (S396) for synthesizing the images cut out by the cut-out process, and an output process (S40) for outputting the images synthesized by the synthesis process as one image data. The cut-out process can remove at least a portion of the area that is not a food storage area from the input captured image (FIG. 11), for example an area in which no food is captured and only the wall surface of the refrigerator or the floor surface outside the refrigerator is captured. By synthesizing the images of the cut-out food storage areas, the ratio of the food storage area displayed in the synthesized image (FIG. 16) can be increased relative to the entire display screen of, for example, the smartphone 55 on which the image is displayed. In this embodiment, the cropping process is performed on one image captured by one imaging device 30, but two or more imaging devices may be prepared to capture two or more images, and the portions including the food storage area included in each image data may be cropped.

As described above, this embodiment is characterized in that the proportion of the area occupied by the food storage area to the entire image data is larger in the image after synthesis processing than in image data captured with a wide-angle lens. For example, comparing Figure 11 with Figure 16, the area showing food is larger, making it easier for the user to see.

In this embodiment, the distance between the multiple parts cut out in the cutting process is shorter after the synthesis process. For example, comparing FIG. 12 with FIG. 16, the distance between the door storage section a, the storage section c, and the door storage section b shown in FIG. 16 is smaller than the distance between the door storage section a, the storage section c, and the door storage section b shown in FIG. 12, making it easier for the user to see. In addition, since multiple storage areas that are separated from each other in actual distance, such as the door storage section and the storage section, and the two door storage sections, are output as one image, the whole can be easily seen without having the user perform a swipe or flick operation to switch the displayed image, as in Patent Document 1. In particular, the two door storage sections are opened by rotating in opposite directions around door hinges arranged at different positions as rotation axes, so the actual distance between them is relatively large in the refrigerator.

In this embodiment, the refrigerator compartment 1 is described as a double-door storage compartment, but it can also be applied to a single-door storage compartment.

<<Second embodiment>>
In the first embodiment, the image synthesis process of the refrigerator compartment 1 (first storage compartment) is described, but the present invention is not limited to this. In the second embodiment, the image synthesis process including the refrigerator compartment 1 (first storage compartment) and the vegetable compartment 4 (second storage compartment) is described.

Figure 17 is a side view showing the state in which the drawer section 6 of the refrigerator 100 according to the second embodiment is pulled out. Figure 18 is an image showing the cropping process of the drawer section 6 of the refrigerator according to the second embodiment. Figure 19 is an image showing the enlargement process of the drawer section 6 of the refrigerator according to the second embodiment. The imaging device 30 for photographing the interior can capture an image of the inside of the drawer section 6. Therefore, the storage state can be understood from the image of the drawer section 6.

Figure 20 is an image showing the synthesis process of each image including the drawer section 6 according to the second embodiment. In Figure 20, compared to Figure 16, an image of the drawer section 6 (second storage compartment d) is also synthesized. This allows the user to grasp the storage status including the refrigerator compartment 1 (first storage compartment) and the vegetable compartment 4 etc. (second storage compartment) at a glance. Note that Figure 16 is an image synthesized from the captured images shown in Figure 12.

That is, the refrigerator 100 is characterized by having at least a first storage compartment that opens like a double door and a second storage compartment that opens in the front-to-back direction, and the cutting process is performed on the first storage compartment and the second storage compartment. The refrigerator 100 may also have at least a first storage compartment that opens like a double door and a second storage compartment that opens in the front-to-back direction, the first storage compartment may have a door storage section, and the cutting process may be performed on the interior of the first storage compartment, the door storage section, and the second storage compartment. The cutting process can be performed on the second storage compartment when it is detected that the second storage compartment is open.

Figure 21 is another example of an image showing the synthesis process of each image including the drawer section 6 according to the second embodiment. Since the display screen of the smartphone 55 on which the image is expected to be displayed is often rectangular, as shown in Figure 21, the positions of the door storage sections a and b can be moved from the side of the storage unit c so that the synthesized image will be rectangular overall. In the example of Figure 21, the door storage sections a and b are moved to the opposite side of the second storage unit d with storage unit c at the center.

That is, the cutting process is performed with at least one door storage section, the interior of the first storage chamber, and the second storage chamber as food storage areas. In the combining process, at least one door storage section and the second storage chamber are arranged to be located on opposite sides of the interior of the first storage chamber.

<<Modifications of the Imaging Device>>
In the first embodiment, the imaging device 30 has been described as having a slide mechanism, but the present invention is not limited to this. In a modified example, a folding mechanism will be described.

Figure 22A is an explanatory diagram showing the normal state of the imaging device 30. Figure 22B is an explanatory diagram showing the state of the imaging device 30 in the middle of being stored. Figure 22C is an explanatory diagram showing the state of the imaging device 30 when stored.

The refrigerator 100 (see FIG. 1) includes a box that forms a storage compartment that is open at the front, an imaging device 30 that is provided outside the storage compartment, and a support section 45 (hinge section) that supports the imaging device 30 so that it can move at least in a direction perpendicular to the opening.

As shown in Figures 22A to 22C, the tip of the imaging device 30 having the camera 33 is configured to be folded in half from the support part 45, and as this does not affect the depth dimension during packaging, the number of refrigerators that can be loaded onto a transport truck is not reduced, and even after installation, when the imaging device 30 is not in use, it can be stored to conceal the imaging range.

The imaging device 30 can also revolve (move in a circular path around the hinge; revolve around an axis) in the direction of looking into the interior of the fridge. Since the box is in the circular path, the angle at which it can revolve is less than 360° when observed from a direction perpendicular to the revolution plane. In this case, a drive unit capable of applying a force in the revolution direction to the imaging device 30 can be attached, and the imaging device 30 can be controlled to be at an angle at which the interior of the fridge or other compartment can be viewed during imaging. For example, the imaging device 30 can be positioned within the movement path of the door during imaging. This allows the interior of the fridge to be imaged more effectively, and since the imaging device can be returned to outside the movement path of the door after imaging is completed, or revolved behind the edge of the front opening of the door or storage compartment for storage, it is characterized by the fact that it is easy to lengthen the arm connecting the support part 45 (hinge part) and the camera.

The present invention is not limited to the first and second embodiments described above, and includes various modified examples. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the configurations described. In addition, it is possible to add, delete, or replace some of the configurations of the above-described embodiments with other configurations. For example, the shooting button 39 in FIG. 3 may be provided on the refrigerator compartment door 1a or on the top surface of the drawer section 6. In addition, multiple imaging devices may be prepared for shooting, and image data of the food storage area contained in each captured image may be used to synthesize into a single image.

According to the refrigerator 100 of this embodiment, curved images captured by an imaging device with a fisheye lens (wide-angle lens) installed on the outside of the storage compartment are distortion-corrected, cropped, and synthesized to output a single image, thereby providing an image of the interior of the refrigerator that is close to the state observed by the user while minimizing image switching operations on the display operation device.

An imaging device with a wide-angle lens such as a fisheye lens installed outside the storage compartment captures an image of the inside of the storage compartment, including the door storage compartment, from above in a bird's-eye view. By correcting distortion of the curved image, cutting out and processing each storage compartment, and synthesizing the images after processing, the situation inside the refrigerator can be presented in a single image as close to the user's line of sight, making it easier to recognize the situation inside the refrigerator. In addition, since there is no need to install multiple imaging devices or imaging devices with complex variable control, it has an advantage in terms of cost, and it can solve problems such as lens scratches and obstruction of the shooting range due to stored items being close to the lens. Users can check the latest situation inside the refrigerator through images even when they are out, which can lead to a reduction in unnecessary shopping and food waste.

1 Refrigerator compartment (first storage compartment)
1a, 1b Refrigerator door 2 Ice making compartment (second storage compartment)
2a Ice maker door 3 Upper freezer (second storage compartment)
3a Upper freezer door 4 Vegetable compartment (second storage compartment)
4a Vegetable compartment door 5 Lower freezer compartment (second storage compartment)
5a Lower freezer door 6 Drawer section 10 Box body 10a Opening edge surface 11a, 11b Door storage section 30 Imaging device 31 Case section 32 Lid section 33 Camera (imaging section)
34 Lens (wide-angle lens)
35 Light shielding wall 38 Lighting unit 39 Photography button (photography command receiving unit)
41 Control SoC (output section)
42 Wireless LAN unit 45 Support part (hinge part)
51 Door sensor 52 Main microcomputer (processing device)
53 Home router 54 Server 55 Smartphone 100 Refrigerator

Claims (4)

A storage chamber having an opening at the front;
a revolving door that opens and closes the opening and has a door storage section;
an imaging device disposed outside the storage chamber and having a lens and an illumination unit located forward of an edge surface of the opening;
a light shielding wall for blocking direct light is provided between the illumination unit and the lens;
The imaging device is arranged so as to be able to include the door storage section in a part of a periphery of an imaging range,
The illumination unit is disposed on the door storage unit side of the imaging system relative to the lens. A storage chamber having an opening at the front;
a revolving door that opens and closes the opening and has a door storage section;
an imaging device disposed outside the storage chamber and having a lens and an illumination unit located forward of an edge surface of the opening;
a light shielding wall for blocking direct light is provided between the illumination unit and the lens;
the imaging device is disposed so as to be able to include the interior of the storage chamber in a part of a periphery of an imaging range;
The illumination unit is disposed on the inner side of the storage chamber relative to the lens. A storage chamber having an opening at the front;
a revolving door that opens and closes the opening and has a door storage section,
The imaging device is disposed outside the storage chamber and has a lens and an illumination unit located forward of an edge surface of the opening,
a light shielding wall for blocking direct light is provided between the illumination unit and the lens;
The door storage section as a food storage area is arranged so as to be included in a part of the periphery of the imaging range,
The illumination unit is an imaging device disposed on the door storage unit side of the lens. A storage chamber having an opening at the front;
a revolving door that opens and closes the opening and has a door storage section,
The imaging device is disposed outside the storage chamber and has a lens and an illumination unit located forward of an edge surface of the opening,
a light shielding wall for blocking direct light is provided between the illumination unit and the lens;
The imaging range is arranged so as to include the interior of the storage chamber as a food storage area in a part of the periphery of the imaging range,
The illumination unit is an imaging device disposed on the inner side of the storage chamber relative to the lens.

Images