JP7583752B2 - Image management system, image management server, image management method, and method for producing image management system - Google Patents

Description

The present invention relates to an image management system, an image management server, an image management method, and a method for producing an image management system.

In recent years, a service has been provided that allows users to view the interior of a refrigerator on a mobile device by uploading images of the interior of the refrigerator to a server and processing the uploaded images. Patent Document 1 describes an invention that makes it possible to capture images of many locations using a small number of cameras.

JP 2003-42626 A

Images taken of the inside of a customer's refrigerator must be handled with care to protect personal information, as there is a risk that the customer or their living environment may be unintentionally seen by others.

In order to solve the above-mentioned problems, the image management system of the present invention comprises a storage facility having a storage area for storing items and an imaging unit for imaging the area of items including the storage area, a pre-processed image acquisition means for acquiring a pre-processed image captured by the imaging unit, a first memory means for storing the pre-processed image, a processed image creation means for , when a face is detected in the pre-processed image, performing processing to reduce the amount of information in the area in which the face appears, thereby creating a processed image, a second memory means for storing the processed image, and a deletion means for deleting the pre-processed image after the processed image has been created.

The image management server of the present invention is characterized by comprising: a pre-processed image acquisition means for acquiring a pre-processed image captured by an imaging unit of an item range including a storage area for storing items; a first storage means for storing the pre-processed image; a post-processed image creation means for, when a face is detected in the pre-processed image , performing processing to reduce the amount of information in the area in which the face appears , thereby creating a post-processed image; a second storage means for storing the post-processed image; and a deletion means for deleting the pre-processed image at least before the post-processed image is deleted.
The image management system of the present invention is characterized by comprising a storage facility having a storage area for storing items, an imaging unit for imaging an area of items including the storage area, a pre-processed image acquisition means for acquiring a pre-processed image captured by the imaging unit, a first storage means for storing the pre-processed image, an processed image creation means for, when a face is detected in the pre-processed image, performing processing to reduce the amount of information in the area in which the face appears, thereby creating a processed image, and a means for enabling the pre-processed image to be viewed on a terminal used by a user.

The image management method of the present invention is characterized in that one or more calculation units execute the steps of: acquiring a pre-processed image captured by an imaging unit of an item range including an item storage area of a storage shed; storing the pre-processed image; when a face is detected in the pre-processed image, performing processing to reduce the amount of information in the area containing the face to create a post-processed image; storing the post-processed image; and deleting the pre-processed image at least before the post-processed image is deleted.
The method for producing an image management system of the present invention is a method for producing an image management system comprising a storage shed having a storage area for storing items and an imaging unit for imaging the item range including the storage area, a pre-processed image acquisition means for acquiring a pre-processed image captured by the imaging unit, a first memory means for storing the pre-processed image, a processed image creation means for , when a face is detected in the pre-processed image , performing processing to reduce the amount of information in the area in which the face appears, thereby creating a processed image, a second memory means for storing the processed image, and a deletion means for deleting the pre-processed image after the processed image has been created, and a control program for controlling the imaging unit is provided in the storage shed or a terminal used by a user.
A method for producing an image management system of the present invention is a method for producing an image management system comprising: a storage shed having a storage area for storing items, an imaging unit that images an area of items including the storage area, a pre-processed image acquisition means that acquires a pre-processed image captured by the imaging unit, a first memory means that stores the pre-processed image, a processed image creation means that, when a face is detected in the pre-processed image, performs processing to reduce the amount of information in the area in which the face appears, thereby creating a processed image, and a second memory means that stores the processed image, and a control program that enables the processed image to be viewed is provided to a terminal used by a user.

Other means will be described in the description of the invention.

FIG. 1 is a block diagram of a refrigerator according to an embodiment of the present invention. FIG. 2 is a block diagram of two types of servers. FIG. 2 is a block diagram of a mobile terminal. FIG. 2 is a logical block diagram of the image management system. FIG. FIG. FIG. 2 is a front view of the refrigerator with the left and right doors of the refrigerator compartment open. FIG. 2 is a plan view of the refrigerator with the left and right doors of the refrigerator compartment open. This is a perspective view of the camera unit of the refrigerator when viewed diagonally from below. FIG. 11 is a sequence diagram of a first image processing process. FIG. 13 is a diagram showing an unprocessed image captured by a refrigerator camera. FIG. 13 is a diagram showing an unprocessed image in which unnecessary parts are masked. FIG. 13 is a diagram showing an example of a processed image. FIG. 13 is a diagram showing a selection screen displayed on the mobile terminal. FIG. 13 illustrates a warning screen. FIG. 13 is a diagram showing a registration continuation user selection screen. FIG. 13 illustrates a confirmation dialog box. FIG. 13 shows a screen for cautions when transferring. FIG. 13 shows a screen for cautions when transferring. FIG. 13 is a diagram showing an image usage warning screen. FIG. 2 is a diagram showing a home screen. FIG. 13 shows a refrigerator contents screen. FIG. 11 is a sequence diagram of an image processing process according to the second embodiment.

The following describes in detail an embodiment of the present invention with reference to the drawings. In this embodiment, the image management system provides services to users by linking the refrigerator 1 in FIG. 1, the servers 2a and 2b and the storage servers 6a and 6b in FIG. 2, and the mobile terminal 3 in FIG. 3. The refrigerator 1, the servers 2a and 2b, the storage servers 6a and 6b, and the mobile terminal 3 are each assigned a different IP address.

The configuration of a refrigerator 1 will be described with reference to FIG.
The refrigerator 1 has, as its hardware configuration, a camera unit 10, a calculation unit 112, a ROM (Read Only Memory) 113, a RAM (Random Access Memory) 114, a communication unit 115, and a camera LED (Light Emitting Diode) 100. The refrigerator 1 further has a left door 12, a right door 13, a door sensor 117, and a refrigeration compartment 116, as shown in FIG.

The calculation unit 112 is, for example, a CPU (Central Processing Unit) and controls the refrigerator 1. The RAM 114 serves as a working area for the calculation unit 112. The RAM 114 is also a non-temporary or temporary recording medium that stores various programs and data, and stores programs (not shown).

The camera unit 10 is an imaging unit that captures images of the refrigerator compartment 116, and is, for example, a fisheye camera or a panoramic camera. The camera unit 10 is attached to the upper side of the refrigerator compartment 116. The camera LED 100 is an indicator that lights up when the camera unit 10 is capturing an image. The left door 12 and the right door 13 are doors provided in the refrigerator compartment 116, and the open/closed state is detected by the door sensor 117. When the user opens the left door 12 and/or the right door 13, the camera unit 10 operates to capture an image of the refrigerator compartment 116. The refrigerator compartment 116 is a storage facility that stores items, such as food.

The communication unit 115 connects to the network and transmits and receives data to and from the server 2a, which is an external device.

The configuration of the servers 2a, 2b and the storage servers 6a, 6b will be described with reference to FIG. 2. The servers 2a, 2b and the storage servers 6a, 6b each have a CPU 21, a ROM 22, a RAM 23, a storage unit 24, and a communication unit 25.

The CPU 21 controls the servers 2a and 2b. The ROM 22 is a non-volatile memory, and stores, for example, a BIOS (Basic I/O System). The RAM 23 serves as the working area for the CPU 21. The RAM 23 is also a non-temporary or temporary recording medium that stores various programs and data, and stores a program (not shown).

The storage unit 24 is a non-temporary or temporary recording medium that stores various programs and data, and is composed of, for example, a hard disk drive (HDD) or a flash memory. It is preferable that the storage units 24 of the storage servers 6a and 6b have a large capacity and can be accessed at high speed. The communication unit 25 is connected to a network and transmits and receives data to and from external devices.

The configuration of the mobile terminal 3 will be described with reference to FIG. 3. The mobile terminal 3 has a CPU 31, a ROM 32, a RAM 33, a storage unit 34, a communication unit 35, and a touch panel display 36.

The CPU 31 controls the mobile terminal 3. The ROM 32 is a non-volatile memory in which, for example, the BIOS is stored. The RAM 33 serves as a working area for the CPU 31. The RAM 33 is also a non-temporary or temporary recording medium that stores various programs and data, and stores programs (not shown).

The storage unit 34 is a non-temporary or temporary recording medium that stores various programs and data, and is composed of, for example, a flash memory. The communication unit 35 connects to a network and transmits and receives data between external devices. The touch panel display 36 is composed of a touch panel layered on a flat display. The touch panel display 36 is a display means that displays characters, images, or figures, and is also an input means that detects user operations.

FIG. 4 is a logical block diagram of the image management system.
The unprocessed image acquisition means 81 acquires an unprocessed image captured by the camera unit 10. The unprocessed image acquisition means 81 is realized by the CPU 21 of the server 2a executing a program.
The first storage unit 82 stores the unprocessed image. The first storage unit 82 corresponds to the storage server 6a.

The processed image creation means 84 creates a processed image by executing processing on the pre-processing image to delete or reduce the amount of information on at least a portion of the area other than the item area. This processed image creation means 84 creates a processed image by deleting or reducing the amount of information on the pre-processing image, including a predetermined area. And when a person's body part is captured in any of the pre-processing images, the processed image creation means 84 creates a processed image by deleting or reducing the amount of information on the person's body part, including the body part. This processed image creation means 84 is embodied by the CPU 21 of the server 2b executing a program.

The deletion means 83 deletes the pre-processed image after the post-processed image is created. The deletion means 83 deletes the pre-processed image at least before the post-processed image is deleted. This is realized by the CPU 21 of the server 2a executing a program.

The second storage unit 85 stores the processed image. This second storage unit 85 corresponds to the storage server 6a.
The registration means 86 registers users who can view the processed image by linking them to the individual number of the storage or the imaging unit. The registered user information presentation means 87 presents information about already registered users to a new user who is to be linked to an individual number.

When registering a new user to be linked to this individual number, the registered user authority deletion means 88 has the user select a previously registered user and deletes the viewing authority of the selected registered user. The registration means 86, registered user information presentation means 87, and registered user authority deletion means 88 are embodied by the CPU 31 of the mobile terminal 3 executing a program.

FIG. 5 is a front view of the refrigerator 1.
Refrigerator 1 is an appliance for cooling food and the like, and includes a left door 12 and a right door 13 for a refrigerator compartment 116, and a camera unit 10. Refrigerator 1 has a plurality of storage compartments inside its housing. The storage compartments of refrigerator 1 include, from the top, refrigerator compartment 116, an ice-making compartment 14 and an upper freezer compartment 15 arranged on the left and right, a vegetable compartment 16, and a lower freezer compartment 17.

The housing of the refrigerator 1 is constructed by filling a space between an outer box made of steel plate and an inner box (not shown) made of resin with a heat insulating material (not shown) such as urethane foam. The front side (front side) of the housing of the refrigerator 1 is provided with a number of openings corresponding to each compartment. For example, when food or the like is placed through the opening of the refrigerator compartment 116, the left door 12 and the right door 13 are opened. When the left door 12 and the right door 13 are closed, the opening of the refrigerator compartment 116 is blocked. In this way, the left door 12 and the right door 13 have the function of blocking the opening of the refrigerator compartment 116. The same applies to the other doors.

The refrigerator 1 has a left door 12 and a right door 13 as French-style doors that form the refrigerator compartment 116 together with the housing. The left door 12 is rotatable around the axis of a hinge 121 (see FIG. 8) at the left end. The same is true for the right door 13. The refrigerator 1 also has a vegetable compartment door and a lower freezer door as pull-out doors in addition to an ice-making compartment door and an upper freezer compartment door.

The refrigerator compartment 116 is provided with a number of shelves that divide the refrigerator compartment 116 into predetermined sections. The inside of the left door 12 is provided with a number of door pockets 123 (see FIG. 7) for storing food and the like. The same is true for the left door 12. The ice-making compartment 14 is provided with an ice-making compartment container (not shown) that is pulled out together with the ice-making compartment door. Similarly, the upper freezer compartment 15 is provided with an upper freezer compartment container (not shown), the vegetable compartment 16 is provided with a vegetable compartment container (not shown), and the lower freezer compartment 17 is provided with a lower freezer compartment container (not shown).

Although not shown, the refrigerator 1 also includes a compressor, a radiator (condenser), a capillary tube (throttle mechanism), and an evaporator. The refrigerant circulates through the compressor, radiator, capillary tube, and evaporator in sequence, and the air in the storage compartment is cooled by heat exchange with the refrigerant flowing through the evaporator. The camera unit 10 shown in FIG. 5 is used to capture images of the storage compartment of the refrigerator 1, and is installed on the top surface of the housing.

FIG. 6 is a side view of the refrigerator 1.
As shown in Fig. 6, the camera unit 10 includes a main body 101 and a support 102. The main body 101 has a function of capturing an image of a storage compartment, etc. The support 102 supports the main body 101 and is installed on the top surface of the housing of the refrigerator 1.

A lens 103 (see also FIG. 6) is provided near the front end of the main body 101. The lens 103 is an optical element that refracts and focuses light. For example, a fisheye lens is used as such a lens 103. When the left door 12 and/or the right door 13 (see FIG. 5) are opened, the lens 103 faces downward so that the camera unit 10 can capture an image of the refrigerator compartment 116 (see FIG. 7). The camera unit 10 is attached in a position where it is possible to capture an image of the living space range, including the living space outside the refrigerator compartment 116.

As shown in FIG. 6, the lens 103 is located forward of the front end of the housing (the opening of the housing). More preferably, the lens 103 is located further forward than the front of the left door 12 and right door 13 (see FIG. 5) in the closed state. This makes it easier for the lens 103 to see the refrigerator compartment 116 when the left door 12 and right door 13 of the refrigerator compartment 116 are opened, for example. It is also possible to capture images of the vegetable compartment 16, etc. (see FIG. 5) with the camera unit 10 when a drawer-type door such as the vegetable compartment door is open. In this way, the lens 103 is provided in a position that allows a view down on each storage compartment.

FIG. 7 is a front view of refrigerator 1 with left door 12 and right door 13 open.
As described above, the inner panel 122 of the left door 12 is provided with a plurality of door pockets 123 for storing food and the like (the same is true for the right door 13). When the left door 12 and the right door 13 are opened, the food and the like in the refrigerator compartment 116 and the door pockets 123, 133 are arranged to be viewed from above within the field of view of the lens 103 of the camera unit 10.

The rotating partition 124 shown in FIG. 7 is a partition for preventing cold air from leaking from the gap between the left door 12 and the right door 13. In the example of FIG. 7, the rotating partition 124 is provided at the end of the left door 12 opposite the axis of the hinge 121 (see FIG. 8), and is designed to rotate in a predetermined manner when the left door 12 is opened or closed.

When manually photographing the refrigerator compartment 116 using the camera unit 10, a photographing button 125 to be pressed by the user is provided on the left door 12, and a photographing button 135 is provided on the right door 13. In the example of FIG. 7, the photographing button 135 is provided on the lower part of the surface of the left door 12 opposite the hinge 121 (see FIG. 8) of the left door 12 (similar to the right door 13). From another perspective, when the left door 12 and the right door 13 are closed (see FIG. 5), the photographing button 125 is provided on the surface of one of the left door 12 and the right door 13 facing the other (near the rotating partition 124 in FIG. 7).

Photographing performed when at least one of these photographing buttons 125, 135 is pressed by the user is called "manual photographing." Photographing performed by the calculation unit 112 based on the elapsed time from the timing when the left door 12 and the right door 13 start to open is called "automatic photographing."

FIG. 8 is a plan view of the refrigerator 1 with the left door 12 and the right door 13 open.
In the example of Fig. 8, the main body 101 of the camera unit 10 is disposed slightly to the left of the center of the housing of the refrigerator 1 in the left-right direction. To explain in more detail, the main body 101 of the camera unit 10 is disposed directly above the seam between the left door 12 and the right door 13 when the door is closed (see also Figs. 5 and 6). By disposing the main body 101 in this manner, for example, when the left door 12 and the right door 13 are opened, food and the like in the door pockets 123, 133 (see Fig. 7) are more likely to come into the field of view of the camera unit 10.

When using the refrigerator 1, the user is often conscious of the seam (boundary) between the left door 12 and the right door 13. Therefore, by positioning the main body unit 101 directly above the seam between the left door 12 and the right door 13, the user is less likely to feel uncomfortable compared to when the main body unit 101 is positioned left-right away from the seam.

FIG. 8 is a perspective view of the camera unit 10 when viewed obliquely from below.
The main body 101 of the camera unit 10 shown in Fig. 8 includes the lens 103 (see also Fig. 6) described above, as well as a case 31b and a camera LED 100 (Light Emitting Diode: illumination unit). In the example of Fig. 6, the case of the main body 101 is generally a rectangular parallelepiped that is elongated in the front-rear direction. A circular hole (not shown) is provided on the underside near the front end of the case of the main body 101, and the lens 103 is exposed through this hole. The camera LED 100 is provided on the rear side (rear side) of the lens 103.

The camera LED 100 is a light source that irradiates the refrigerator compartment 116 (see FIG. 7) and the door pockets 123, 133 (see FIG. 7) with light so that the camera unit 10 can photograph the refrigerator compartment 116 (see FIG. 7) and the door pockets 123, 133 under appropriate brightness. The camera LED 100 (illumination unit) is provided on the outside of the housing of the refrigerator 1 (see FIG. 6). Details will be described later, but in this embodiment, the camera unit 10 photographs the refrigerator compartment 116 (see FIG. 7) with the interior light of the refrigerator 1 turned off and the camera LED 100 turned on.

As shown in FIG. 9, by providing lens 103 in front of camera LED 100, reflected light from refrigerator compartment 116, etc. (see FIG. 5) can easily enter lens 31a when left door 12 and right door 13 are open. Therefore, food stored in refrigerator compartment 116, etc. can be properly photographed. It is preferable that camera LED 100 is positioned forward of the front end (opening of the housing) of refrigerator 1's housing (see FIG. 6). This makes it easier for light emitted from camera LED 100 to enter refrigerator compartment 116, etc., and ultimately makes it easier to obtain clear photographic results.

Refrigerator 1 is a storage unit that includes a refrigerator compartment 116, which is a storage area for storing items, and a camera unit 10 that captures images of the area containing the items, including the refrigerator compartment 116.

FIG. 10 is a sequence diagram of the image processing according to the first embodiment.
When the user opens the door of the refrigerator 1, the calculation unit 112 detects that the door has been opened by the door sensor 117. This causes the calculation unit 112 to capture an image of the refrigerator compartment 116 in which food is stored by the camera unit 10 (step S10). The calculation unit 112 uploads the captured camera image of the refrigerator compartment 116 to the server 2a (step S11). This camera image is an image before any processing that can be viewed by the user.

The CPU 21 of the server 2a functions as an unprocessed image acquisition means for acquiring an unprocessed image captured by the camera unit 10. The server 2a transmits this unprocessed camera image to the storage server 6a (step S12).

When the storage server 6a notifies the server 2b of the event (step S13), the storage server 6a stores the unprocessed camera image (step S14). The storage server 6a functions as a first storage unit that stores the unprocessed image.
Then, when the storage server 6a transmits the unprocessed camera image to the server 2b (step S15), it deletes the unprocessed camera image stored in itself (step S16). That is, the CPU 21 of the storage server 6a functions as a deletion unit that deletes the unprocessed camera image.

The server 2b corrects the distortion of the camera image before processing due to the lens (step S17) and processes the image into a front view image using keystone correction (step S18). The server 2b then requests the existing processed image from the storage server 6b (step S19) and receives the processed image from the storage server 6b (step S20).

When the server 2b combines the existing processed image with the keystone corrected image (step S21), it transmits the combined processed image to the storage server 6b and stores it (step S22). The CPU 21 of the server 2b functions as a processed image creation means that performs processing on the pre-processing image to delete at least a part of the area other than the item area or reduce the amount of information, and creates a processed image.

The storage server 6b stores the processed image received from the server 2b (step S23). The storage server 6b functions as a second storage unit that stores the processed image.
Then, in response to a request from the mobile terminal 3, the storage server 6b transmits the processed image to the mobile terminal 3 (step S24). This allows the customer to view the image of the refrigerator compartment 116 on the mobile terminal 3.

By storing the unprocessed image in storage server 6a and storing the synthesized processed image in storage server 6b, it becomes easier to manage each image. The mobile terminal 3 can easily obtain the synthesized processed image by simply accessing storage server 6b.

The mobile terminal 3 does not access the storage server 6a, and therefore does not know the IP address of the storage server 6a. This prevents the mobile terminal 3 from knowing how to access the storage server 6a, and therefore prevents the customer or the customer's living environment from being seen unintentionally by others in the unedited image.

When a predetermined period of time has elapsed since the storage server 6b saved the edited image, the storage server 6b deletes the edited image. In other words, before the storage server 6b deletes the edited image, the storage server 6a deletes the unedited image that it saved. This makes it possible to prevent a customer who appears in the unedited image or the customer's living environment from being unintentionally seen by others.

Figure 11 shows an image taken by the camera unit 10 of the refrigerator 1 before processing. The lower half of the image shows the refrigerator compartment 116. The left side of the image shows the inside of the left door 12. The right side of the image shows the inside of the right door 13. The upper half of the image shows the customer's living environment. This part is unnecessary when viewed from the processed image.

FIG. 12 is a diagram showing the image before processing in which the unnecessary parts have been painted out.
The system performs a masking process on unnecessary parts of the fisheye image. The masking process performed by the system is a processing process that deletes at least a part of the area other than the area of the item. This prevents the customer and their living environment from being unintentionally seen by others when viewing an image of the inside of the customer's refrigerator.
The masking process of the unnecessary parts is executed by the server 2b, but may be executed by the server 2a or the refrigerator 1.

FIG. 13 is a diagram showing an example of the processed image.
The server 2b corrects lens distortion of the unprocessed image captured by the camera unit 10 of the refrigerator 1 and processes it into a front view image by performing keystone correction. At this time, if the server 2b detects a face in the image, it performs mosaic processing on the area in which the face appears.

The pixelization process performed by this system is a processing process that reduces the amount of information in at least a portion of the area other than the area of the items. This prevents the customer from unintentionally letting others see the image of the inside of the refrigerator when the image is viewed. Note that this face detection process and pixelization process are performed by server 2b, but server 2a or refrigerator 1 may also perform them on the image before processing.

Figures 14 to 20 show a series of images for confirming with the customer important points regarding the handling of camera images when pairing the mobile terminal 3 with the refrigerator 1. These screens are displayed by the CPU 31 of the mobile terminal 3 executing the refrigerator management program installed in the mobile terminal 3.

FIG. 14 is a diagram showing a new/used item selection screen 41 displayed on the mobile terminal 3.
The selection screen 41 includes a new purchase button 411 , an other button 412 , and a back button 419 .

The new purchase button 411 is for users who have made a new purchase. When this button is tapped, the mobile terminal 3 pairs with the refrigerator 1 and then registers the user.

The Other button 412 is a button for those who have purchased a second-hand refrigerator 1. When this button is tapped, the mobile terminal 3 transitions to a warning screen 42 of FIG.
The back button 419 is a button for transitioning to the previous screen before this selection screen 41 .

FIG. 15 is a diagram showing the warning screen 42. As shown in FIG.
The warning screen 42 is a screen that warns about points to be careful about when the refrigerator 1 is a second-hand item.
The confirmation button 421 is a button for ending the confirmation on the warning screen 42. When the confirmation button 421 is tapped, the mobile terminal 3 pairs with the refrigerator 1 and then performs user registration.
The back button 429 is a button for transitioning to the previous screen before the warning screen 42 .

By registering the user, the identification number (individual number) of the refrigerator 1 is linked to the mobile terminal 3. Note that the identification number (individual number) of the refrigerator 1 is not limited, and the individual number of the camera unit 10 may be linked to the mobile terminal 3 for user registration.

If there is already a registered user after the user registration, the screen transitions to the continued registration user selection screen 43 in FIG. 16. If there is no already registered user after the user registration, the screen transitions to the transfer warning screen 45 in FIG. 18.

FIG. 16 is a diagram showing the registration continuation user selection screen 43. As shown in FIG.
The continued registration user selection screen 43 is a screen for selecting a user whose registration to this refrigerator 1 is to be continued. Check boxes 431 to 433 are check boxes for selecting a user whose registration is to be continued.

The Confirm button 435 is a button for confirming the selection of a user for continuing registration after the selection. Tapping the Confirm button 435 causes the mobile terminal 3 to transition to the confirmation dialog box 44 in FIG. 17. The Back button 439 is a button for transitioning to the previous screen, which is the continued registration user selection screen 43.

FIG. 17 is a diagram showing the confirmation dialog box 44 .
The confirmation dialog box 44 is a dialog for confirming the deregistration of a user whose registration will not continue.

The Yes button 441 is a button for deleting the registration of users who have not been selected.
The No button 442 is a button for not deleting the registration of the non-selected user. When the Yes button 441 or the No button 442 is tapped, the screen transitions to the transfer notice screen 45 in FIG.

FIG. 18 is a diagram showing a transfer warning screen 45.
The transfer warning screen 45 is a screen that displays warnings to be observed when transferring this refrigerator 1. The confirmation button 451 is a button that indicates that the warnings have been confirmed.
The back button 459 is a button for transitioning to the previous screen, which is the transfer warning screen 45 .

FIG. 19 is a diagram showing the camera usage warning screen 46. As shown in FIG.
The camera usage warning screen 46 is a screen for alerting the customer to taking pictures with the camera unit 10, which is the camera of the refrigerator 1. When the customer swipes this screen and reaches the bottom, a confirmation button for transitioning to the next image usage warning screen 47 is displayed.
The back button 469 is a button for transitioning to the previous screen, ie, the camera usage warning screen 46 .

FIG. 20 is a diagram showing the image usage warning screen 47.
The image utilization notice screen 47 is a screen for obtaining the customer's consent regarding utilization of images taken by the camera unit 10 which is the camera of the refrigerator 1.
The consent button 471 is a button for the customer to consent to the use of the image. When the consent button 471 is tapped, a series of settings is completed and the screen transitions to the home screen 50.
The back button 479 is a button for transitioning to the previous screen, before transitioning to the image utilization notice screen 47 .

FIG. 21 is a diagram showing the home screen 50. As shown in FIG.
The home screen 50 is a screen that is displayed after the refrigerator management program is started.
The check refrigerator contents button 501 is a button for viewing the contents stored in the customer's refrigerator 1. When the check refrigerator contents button 501 is tapped, the mobile terminal 3 transitions to a refrigerator contents screen 51 of FIG.
The usage concierge button 502 is a button that explains how to use the refrigerator 1.

FIG. 22 is a diagram showing the refrigerator contents screen 51.
The refrigerator contents screen 51 is a screen for viewing a front view image of the refrigerator compartment 116 of this refrigerator 1.
The room selection button 512 is a button for selecting whether to display the refrigerator room 116, the inside of the left door 12, or the inside of the right door 13.
The back button 519 is a button for transitioning to the home screen 50, which is the screen immediately before transitioning to the refrigerator contents screen 51.

Second Embodiment
In the second embodiment, lens correction processing, keystone correction processing, and the like are performed by the calculation unit 112 of the refrigerator 1, not by the server 2b. Since it is not necessary to transmit unprocessed camera images over the network, it is possible to prevent the customer or the customer's living environment from being unintentionally seen by others.

FIG. 23 is a sequence diagram of the image processing according to the second embodiment.
When the user opens the door of the refrigerator 1, the calculation unit 112 detects that the door has been opened by the door sensor 117. In response, the calculation unit 112 captures an image of the refrigerator compartment 116 in which food is stored by the camera unit 10 (step S30). This camera image is an image before any processing is performed that can be viewed by the user.

The calculation unit 112 saves the unprocessed camera image (step S31). The calculation unit 112 corrects distortion caused by the lens in the unprocessed camera image (step S32) and processes the image into a front-view image using keystone correction (step S33). The calculation unit 112 then deletes the unprocessed camera image (step S34). The calculation unit 112 requests the existing processed image from the storage server 6b (step S35) and receives the processed image from the storage server 6b (step S36).

When the calculation unit 112 combines the existing processed image with the keystone corrected image (step S37), it sends the combined processed image to the server 2b (step S38). The server 2b sends the processed image to the storage server 6b for storage (step S39). The storage server 6b stores the processed image received from the server 2b (step S40). Then, in response to a request from the mobile terminal 3, the storage server 6b sends the processed image to the mobile terminal 3 (step S41). This allows the customer to view the image of the refrigerator compartment 116 on the mobile terminal 3.

When a predetermined period of time has elapsed since the storage server 6b saved the edited image, the storage server 6b deletes the edited image. In other words, before the storage server 6b deletes the edited image, the storage server 6a deletes the unedited image that it saved. This makes it possible to prevent a customer who appears in the unedited image or the customer's living environment from being unintentionally seen by others.

The present invention is not limited to the above-described embodiments, and includes various modified examples. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and are not necessarily limited to those having all of the configurations described. It is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. It is also possible to add, delete, or replace part of the configuration of each embodiment with other configurations.

The above configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware such as an integrated circuit. The above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program that realizes each function. Information such as the programs, tables, and files that realize each function can be stored in a recording device such as a memory, a hard disk, or an SSD (Solid State Drive), or on a recording medium such as a flash memory card or a DVD (Digital Versatile Disk).

In each embodiment, the control lines and information lines are those that are considered necessary for the explanation, and not all control lines and information lines in the product are necessarily shown. In reality, it may be considered that almost all components are connected to each other.
As modified examples of the present invention, for example, the following (a) to (d) are available.

(a) The pixelation process to prevent the customer and their living environment from being unintentionally seen by others may be performed by either the refrigerator 1, the server 2a or the server 2b.
(b) The masking process for preventing the customer and the customer's living environment from being unintentionally seen by others may be executed by the refrigerator 1, the server 2a, or the server 2b.
(c) Either the mosaic process or the masking process may be performed, or both may be performed.

(d) The present invention is not limited to viewing the refrigerator compartment of a refrigerator, but may be applied to viewing the storage area of any storage unit.

1 Refrigerator (storage room)
100 Camera LED
10 camera unit 112 calculation unit 113 ROM
114 RAM
115 Communication unit 116 Refrigerator compartment (storage area)
117 Door sensor 12 Left door 124 Rotating partition 13 Right door 14 Ice making compartment 15 Upper freezer compartment 16 Vegetable compartment 17 Lower freezer compartment 2a Server (unprocessed image acquisition means, deletion means)
2b Server (processed image creation means)
21 CPU
22 ROM
23 RAM
24 Storage unit 25 Communication unit 3 Portable terminal 31 CPU
32 ROM
33 RAM
34 Storage unit 35 Communication unit 36 Touch panel display 41 Selection screen 411 New purchase button 412 Other button 419 Back button 42 Warning screen 429 Back button 43 Registration continuation user selection screen 431 Check box 432 Check box 433 Check box 435 Confirm button 439 Back button 44 Confirmation dialog box 441 Yes button 442 No button 45 Transfer warning screen 451 Confirm button 459 Back button 46 Camera use warning screen 469 Back button 47 Image utilization warning screen 479 Back button 50 Home screen 501 Check contents of refrigerator button 502 How to use concierge button 51 Refrigerator contents screen 512 Room selection button 6a Storage server (first storage means)
6b Storage server (second storage means)
81 Unprocessed image acquisition means 82 First storage means 83 Deletion means 84 Processed image creation means 85 Second storage means 86 Registration means 87 Registered user information presentation means 88 Registered user authority deletion means

Claims (9)

a storage area for storing items;
A storage unit including an imaging unit that images an item range including the storage area;
an unprocessed image acquisition means for acquiring an unprocessed image captured by the imaging unit;
A first storage means for storing the unprocessed image;
a processed image creating means for, when a face is detected in the unprocessed image , executing processing to reduce an amount of information in an area in which the face is captured , and creating a processed image;
A second storage means for storing the processed image;
a deletion means for deleting the pre-processed image after the pre-processed image is created;
An image management system comprising: a storage unit having a storage area for storing items;
An imaging unit that images an item range including the storage area;
an unprocessed image acquisition means for acquiring an unprocessed image captured by the imaging unit;
A first storage means for storing the unprocessed image;
a processed image creating means for, when a face is detected in the unprocessed image , executing processing to reduce an amount of information in an area in which the face is captured , and creating a processed image;
A means for enabling a terminal used by a user to view the processed image;
An image management system comprising: a registration means for registering users who can view the processed image by linking the users to individual numbers of the storage unit or the imaging unit;
a registered user information presenting means for presenting information on already registered users to a new user to be linked to the individual number;
3. The image management system according to claim 1, further comprising: a registered user authority deleting means for, when registering a new user to be linked to the individual number, having the user select an already registered user and deleting the browsing authority of the selected registered user;
The image management system according to claim 3, further comprising: The IP address assigned to the first storage means is different from the IP address assigned to the second storage means.
2. The image management system according to claim 1, an unprocessed image acquisition means for acquiring an unprocessed image captured by an imaging unit of an item range including a storage area for storing items;
A first storage means for storing the unprocessed image;
a processed image creating means for, when a face is detected in the unprocessed image , executing processing to reduce an amount of information in an area in which the face is captured , and creating a processed image;
A second storage means for storing the processed image;
a deletion means for deleting the unprocessed image at least before the processed image is deleted;
An image management server comprising: One or more computing units,
A step of acquiring an unprocessed image captured by an imaging unit of an item range including an item storage area of the storage unit;
storing the unprocessed image;
a step of, when a face is detected in the unprocessed image, executing processing for reducing an amount of information in an area in which the face is captured , thereby generating an processed image;
storing the processed image;
deleting the pre-processed image at least before the post-processed image is deleted;
2. An image management method comprising: a storage area for storing items;
A storage unit including an imaging unit that images an item range including the storage area;
an unprocessed image acquisition means for acquiring an unprocessed image captured by the imaging unit;
A first storage means for storing the unprocessed image;
a processed image creating means for, when a face is detected in the unprocessed image , executing processing to reduce an amount of information in an area in which the face is captured , and creating a processed image;
A second storage means for storing the processed image;
a deletion means for deleting the pre-processed image after the pre-processed image is created;
A method of producing an image management system comprising:
A method for producing the image management system by providing a control program for controlling the imaging unit to the repository or a terminal used by a user. a storage unit having a storage area for storing items;
An imaging unit that images an item range including the storage area;
an unprocessed image acquisition means for acquiring an unprocessed image captured by the imaging unit;
A first storage means for storing the unprocessed image;
a processed image creating means for, when a face is detected in the unprocessed image , executing processing to reduce an amount of information in an area in which the face is captured , and creating a processed image;
A second storage means for storing the processed image;
A method of producing an image management system comprising:
A method for producing an image management system by providing a control program that enables a user to view the processed image to a terminal used by the user .

Images