JP7633971B2 - Food Management System - Google Patents

Description

The present invention relates to a food ingredient management system.

In recent years, advances in IT technology have made it possible to manage and control home electrical appliances using the voice input function of mobile information terminals. For example, in relation to refrigerators, Patent Document 1 proposes, with the aim of providing an information processing system that can improve usability in managing ingredients stored in a refrigerator, that "the information processing system has a processing unit that can register multiple types of ingredients before one reception of the voice dialogue function is completed. The processing unit can execute a first mode that ends the activated state when one instruction is received in one startup, and a second mode that continues the activated state even if one instruction is received in one startup, and executes the second mode for registering ingredients."

JP 2021-128559 A

According to Patent Document 1, continuous voice input is possible without having to use a wake-up word for the mobile information terminal every time, and for unknown information such as the quantity associated with the food name, a prompt is displayed to encourage manual input.

However, with voice input, there is a possibility of unintended ingredient names being recognized due to voice recognition errors, and there is room for improvement in smoothly correcting these while registering ingredients.

In view of the above, the present invention provides an ingredient management system that causes one or more computers to operate as an initiation unit that starts accepting voice input, a display output unit that displays a voice-recognized character string on a display screen, an edit processing unit that accepts editing of the character string displayed on the display screen in response to user operation and, if edited, displays the edited character string on the display screen, a termination assignment unit that assigns a terminator to the voice-recognized character string and then treats the voice-recognized character string as a new character string, and a registration processing unit that, in response to user operation, adds the character string displayed on the display screen as an ingredient name to an ingredient list , and further has any of the following configurations (1) to (3) .
(1) The display output unit displays two or more character strings, and the editing processing unit is capable of editing each of the character strings displayed by the display output unit. The registration processing unit adds each of the character strings displayed by the display output unit to an ingredient list as an ingredient name and is capable of referencing a master database that stores ingredient names and ingredient detail information corresponding to the ingredient names. When the registration processing unit operates, the display output unit searches the master database for ingredient names corresponding to each of the character strings displayed, and for a character string that has a corresponding ingredient name, the ingredient name and the corresponding ingredient detail information can be added to the ingredient list as a set. If the master database is searched using the character string displayed by the display output unit as a search word and multiple candidate ingredients that are exact matches are obtained, even if the registration processing unit is operated, the ingredient detail information is not set for the ingredient name.
(2) The display output unit displays a character string and, after the operation of the registration processing unit, automatically operates the start unit and is capable of referring to a master database that stores ingredient names and ingredient detail information corresponding to the ingredient names. The display output unit searches the master database for an ingredient name corresponding to the character string displayed by the display output unit. If a corresponding ingredient name is found, the display output unit displays it as a candidate ingredient. When a selection of a candidate ingredient displayed on the display screen is accepted from a user, the registration processing unit adds a pair of the ingredient name of the candidate ingredient and the corresponding ingredient detail information to an ingredient list. When the master database is searched using the character string displayed by the display output unit as a search word and multiple candidate ingredients that are exact matches are obtained, the ingredient detail information is not set for the ingredient name even if the registration processing unit is operated.
(3) Depending on the presence of ingredients included in the ingredient list for which time has passed without an estimated consumption date being set, or the number of ingredients for which an estimated consumption date has not been set, an output is provided to encourage the user to set an estimated consumption date.

1 is a diagram showing an example of the overall configuration of a foodstuff management device according to an embodiment of the present invention. FIG. 4 is a diagram showing an example of a screen configuration of the portable information terminal 2 when voice input is performed. FIG. 13 is a diagram showing an example of screen transition in method 1. 13A and 13B are diagrams showing an example of screen transition when candidate information exists in method 1. 13A and 13B are diagrams showing an example of screen transition when candidate information does not exist in method 1. FIG. 13 is a sequence diagram showing a series of processing steps among a user, a portable information terminal, and a server in method 1. FIG. 13 is a diagram showing an example of screen transition in method 2. FIG. 13 is a diagram showing an example of screen transition in method 2. FIG. 13 shows an example of a screen transition when changing an ingredient name from a provisional list display. FIG. 13 shows an example of screen transition when deleting an ingredient name from the provisional list display. FIG. 11 is a sequence diagram showing a series of processing steps among a user, a portable information terminal, and a server in method 2. FIG. 2 is a diagram showing an example of the configuration of a master database and a concierge database. FIG. 13 is a diagram showing a schematic flow when referring to a master database and a concierge database. FIG. 13 is a diagram showing an example of a data structure of a concierge database. A diagram showing the assumed storage chamber locations for each model and setting. FIG. 13 is a diagram showing the screen transition when referring to a concierge database. FIG. 13 is a sequence diagram showing a procedure for linking a master database and a concierge database. A diagram showing the process when there are multiple ingredients that exactly match the search word. A diagram showing how to handle the labeling of food ingredients that have passed their recommended consumption date. FIG. 13 is a diagram showing how to handle an ingredient that has been deleted from the list of ingredients being stored. FIG. 13 is a diagram showing an example of a top screen displayed immediately after starting an ingredient management function. 13A to 13C are diagrams showing screen transitions relating to a reminder function for registering estimated consumption. FIG. 13 shows screen transitions when the user can select a method for adding ingredients.

The following describes an embodiment of the present invention with reference to the accompanying drawings. In the embodiment, a foodstuff management device (foodstuff management system) is described, but the present invention can be applied to any item, not limited to foodstuffs.

Fig. 1 is a diagram showing an example of the overall configuration of a foodstuff management device according to an embodiment of the present invention. The foodstuff management device includes a mobile information terminal 2 as a computer, and a server 3 as a computer that stores a master database 31, an ingredient list 32, and a concierge database 33. These are linked via a cloud or a network (hereinafter simply referred to as cloud 5) to perform computer processing and generate a screen for the mobile information terminal 2 as appropriate. For example, the ingredient list 32 may be stored in the mobile information terminal 2, etc. A refrigerator 4 as an example of a storage facility can at least be connected to the cloud 5 if it is equipped with a camera, which will be described later.

As shown in FIG. 12, the master database 31 stores a unique ingredient ID 311 and ingredient name 312 in association with a search word 313 and reference ingredient information 310 (estimated consumption period 314, recommended storage location 315, ingredient category 316, and one or more standard image data 317, 318).

The ingredient list 32 that can be displayed on the mobile information terminal 2 is a list in which one or more ingredients are registered, and for each ingredient, an ingredient name 322 and ingredient details 320 (estimated consumption date 324, storage location 325, ingredient category 326, ingredient image 327) are linked and stored in the ingredient list 32. Please refer to FIG. 12 which shows an example of the display screen of the mobile information terminal 2 displaying the ingredient list 32. As will be described later, a link button 72 with the concierge database 33 can be displayed on the display screen of the ingredient details 320 shown in FIG. 12.

The ingredient management device creates and stores an ingredient list 32 of ingredients managed by the user, and sets and stores the ingredient name 322 and ingredient details 320 of each ingredient registered in the ingredient list 32. The ingredient name 322 and ingredient details 320 can be set by referring to the ingredient name 312 and reference ingredient information 310 in the master database 31, or can be set by the user 1 via the mobile information terminal 2.

The mobile information terminal 2 has a calculation unit CPU, a RAM for temporarily storing data, a ROM for recording programs, a communication unit Tr, and other components connected via a bus BUS. The calculation unit CPU is capable of various processes according to the programs recorded in the ROM, but FIG. 1 mainly describes a processing function specialized for voice input. The mobile information terminal 2 can be equipped with a microphone for receiving voice input.

The functions related to voice input include a start unit 21a that starts accepting voice input as a function of the voice input unit 21 related to voice input, and a continuation unit 21b that starts accepting voice input of a new character string without the need for user 1 operation. In addition, the display output unit 22 displays the voice input character string on the display screen, a termination unit 23a that adds an end to one character string as a function of the character conversion unit 23 that converts the input voice into a character string, an edit processing unit 24 that accepts editing of the character string displayed on the display screen in response to the operation of the user 1, and a registration processing unit 25 that registers ingredients corresponding to each displayed character string in the ingredient list 32. In addition, the individual functions are organically combined and executed, and the overall control via the cloud 5, including user requests, is managed by the central processing unit 26. The central processing unit 26 is not essential.

Figure 2 shows an example of the screen configuration of the mobile information terminal 2 during voice input. In this case, the display screen example has a main screen 91 at the top, a character input screen 92 at the bottom, and a voice input status display section 93 and character string display section 94 in the middle, and is preferably configured to include at least these display sections. Also, for example, various menu buttons 96 are located at the bottom.

In addition, on the main screen 91, information is displayed that is changed successively in response to voice input or operational instructions from the user 1. The character input screen 92 is a character input screen using a keyboard, and allows the user 1 to manually input text (character strings). The voice input status display unit 93 uses, for example, a microphone symbol to distinguish between when voice input is active and when voice input is inactive. The character string display unit 94 displays the character string generated in response to character input from the keyboard using the character input screen 92 or voice input from the microphone, either as is or after converting it into kanji, etc.

In the above-described food ingredient management device, in the first embodiment of the present invention, voice input is performed by the mobile information terminal 2 according to the processing procedure shown in FIG. 3 or FIG. 7. This series of processing is executed based on instructions from the central processing unit 26.

The processing procedure shown in FIG. 3 or FIG. 7 is executed by the food ingredient management device shown in FIG. 1. For example, text editing, which will be described later, is realized by the editing processing unit 24 accepting editing of a character string displayed on the display screen of the mobile information terminal 2 in response to an operation by the user 1. Continuous input by voice is realized by the continuous unit 21b of the voice input unit 21 starting to accept voice input of a new character string.

<Method 1: Registering one ingredient at a time>
The process of registering ingredients in the ingredient list 32 using method 1 will be described. Fig. 3 is a diagram showing an example of screen transitions in method 1. Fig. 4 is a diagram showing an example of screen transitions in a case where candidate information exists in method 1. Fig. 5 is a diagram showing an example of screen transitions in a case where candidate information does not exist in method 1.

First, when voice input is started by operating the voice input status display unit 93 or inputting a wake-up word (S10), voice character input is accepted and the recognized character string is displayed on the character string display unit 94. When the end of voice input is detected, for example, when there is silence for more than one second or the voice input status display unit 93 is tapped (S11, Y), it is determined that the voice input (provisional input) of one ingredient name has ended and that this is the end of one character string (ingredient name). The character string that is voice recognized after the terminator is added can be treated as a new character string. If no registration instruction, described below, is given between the time when the terminator is added and the time when new voice recognition is started, and if there is only one candidate ingredient name found by searching the master database 31, the candidate ingredient name may be registered in the ingredient list 32, or the character string that was input immediately before may be discarded.

Next, the display of the voice input status display unit 93 is set to voice reception OFF, and reception of voice input is temporarily suspended. Also, at least up to this point, the editing processing unit 24 is enabled and a keyboard is displayed on the character input screen 92. In Figures 4 and 5, the editing processing unit 24 is enabled before reception of voice input, but this may also be the case.

Next, the master database 31 is searched using the character string displayed in the character string display section 94 as the search word 313 (S12). If an ingredient (ingredient ID 311 or ingredient name 312) that matches the search word 313 is found (S13, Y), the ingredient name 312 is displayed as a candidate ingredient name, for example, on the main screen 91 (S14).

If the voice recognition is not working properly, the user can edit the character string displayed in the character string display unit 94 as the voice recognition result through the editing processing unit 24. If the character string is edited (S15, Y), the process returns to S12 again with the edited character string. This makes it easy to correct any problems with the voice recognition results.

When a candidate ingredient name displayed on the main screen 91 is selected by tapping or the like (S16, Y), it is determined that a registration instruction has been issued using the selected ingredient name 312. That is, the ingredient name 312 is added to the ingredient list 32 as the ingredient name 322, and the reference ingredient information 310 stored in the master database 31 is automatically set to the ingredient (S17). On the other hand, if a registration instruction is issued by operating, for example, the "OK" button on the keyboard when no candidate ingredient is found in the master database 31, the reference ingredient information 310 cannot be used, so the food details information 320 is not automatically set, and the ingredient name entered in the character string display unit 94 is added to the ingredient list 32. After adding one ingredient to the ingredient list 32 in this way, voice input is started again (S10). This start may be performed automatically as an operation of the continuation unit 21b, or may be performed by operating the voice input status display unit 93.

If there is no ingredient in the master database 31 that matches the search word 313 in S13, the candidate ingredient (ingredient name 312) is not displayed (S13N). In this case, although not shown in FIG. 3, the user may edit the string (S15) and search the master database 31 again to find candidate ingredients (S12). Alternatively, the ingredient name that is not stored in the master database 31 may be added to the ingredient list 32 as is (the string displayed in the string display section 94). In this case, the ingredient is added to the ingredient list 32 as the ingredient name 322, but since there is no reference ingredient information 310 to refer to, the detailed ingredient information 320 is not automatically set.

S12 may also be performed before the end of voice input.

Figures 4 and 5 show examples of screen transitions in each state in the first method, with Figure 4 showing an example of a screen transition when candidate information exists (S13, Y) and Figure 5 showing an example of a screen transition when candidate information does not exist (S13, N). Both show screen transitions from S10.

First, let us explain with reference to Figure 4. The screen display content changes sequentially from screen 101 to screen 105.

Screen 101 can be displayed immediately before S10 (the "Start" state in FIG. 3). The main screen 91 can display a portion of the ingredient list that lists ingredients that have been registered at this point. When the voice input status display section 93 is tapped, the screen transitions to screen 102 (S10).

A keyboard is displayed as a character input screen 92 in a part of the screen 101, for example in the lower part. This allows for both voice input and text editing.

Screen 102 is the screen during voice input, and the voice input status display section 93 is highlighted to indicate that voice input is in progress. The character string confirmed by the voice input is displayed in the character string display section 94. In this example, the user speaks "tomato," and the voice is recognized as intended.

Screen 103 is the screen after voice input (S11, Y). Based on the voice input, the master database 31 is referenced, and "tomato" and then "cherry tomato" are displayed on the main screen 91 as candidates corresponding to the voice input (S14). If the voice recognition result is not good and a character string such as "Tometou" is displayed in the character string display section 94, it can be edited using the character input screen 92. Note that in this state, voice input from the microphone is stopped (S12), so the voice input status display section 93 shows a display indicating that voice input is disabled. Also, while the candidate ingredient names are being displayed, the ingredient displays in the ingredient list 32 are hidden. This makes it easy for the user to recognize that candidate ingredients are being displayed.

Screen 104 is a candidate confirmation screen, and allows the user to instruct (select) one of the ingredients displayed on main screen 91 by tapping, etc. In method 1, for example, tapping on the name of a candidate ingredient functions as registration instruction section 95. Here, it is assumed that "tomato" has been selected and confirmed.

Screen 105 is the screen after the registration instruction. The selected ingredient "tomato" has been added to the ingredient list 32 on the main screen 91. At this time, the ingredient information for "tomato" is not added to the ingredient list 32 only with the ingredient name 322 "tomato", but the ingredient details 320 are automatically set by referencing the reference ingredient information 310 set in the master database 31. For example, using the date 329 (see FIG. 12) of "tomato" added to the ingredient list 32 as a reference, the number of days set in the estimated consumption period 314 is referenced, and the date this number of days after the date 329 is automatically set as the estimated consumption date 324 for "tomato". Also, for example, from the item in the master database 31 where the ingredient name 312 is "tomato", the recommended storage location 315 "vegetable compartment" is referenced and set as storage location 325.

Now, on screen 105, after "tomato" is added to the ingredient list, voice input is automatically activated (S10). The voice input status display unit 93 is highlighted to indicate that voice input is in progress. In this way, in method 1, which accepts instructions to register one ingredient at a time, acceptance of voice input is temporarily halted after the end of the character string is added (S12). The editing processing unit 24 operates in response to instructions from user 1, and then the registration instruction unit 95 operates to add one ingredient to the ingredient list 32, after which the continuation unit 21b operates to start accepting voice input of a new character string. The next ingredient can be entered immediately.

Next, FIG. 5 will be described. The screen display contents change in sequence from screen 111 to screen 115. Note that screens 111 and 112 are basically the same as screens 101 and 102 in FIG. 4, and therefore the description will not be repeated. However, it is assumed that screen 112 has been entered with "〇△□×", which is a character string that does not have a matching search word 313 registered in the master database 31.

When the master database 31 is referenced on the screen 113 using the character string in the character string display section 94 as the search word 313, no matching candidate ingredients are found (no ingredient names 312 linked to the search word 313 are found), so no candidates are displayed on the main screen 91 (S13N).

Screen 114 shows a scene in which the "Done" button as the registration instruction section 95 is selected by keyboard operation on the character input screen 92 when no candidate ingredients have been found.

Screen 115 is the screen after the registration instruction has been given, and an ingredient has been added with the character string "〇△□×" that was entered on main screen 91 as ingredient name 322. However, because no corresponding candidate ingredient has been found in master database 31, food detail information 320 such as estimated consumption date 324 and storage location 325 cannot be automatically set. In other words, only the ingredient name 322 is registered, and ingredient detail information 320 is registered as not set (S17). Then, voice input acceptance begins again (S10).

In this way, if no candidate information exists, the image 327 and estimated consumption date 324 of the ingredient are not automatically set. For an ingredient "〇△□×", the user can select the ingredient from the ingredient list 32, edit it, and manually set the ingredient details 320.

Figure 6 is a sequence diagram showing a series of processing steps between user 1, mobile information terminal 2, and server 3 in the first method. In processing step S201, user 1 taps the microphone button (corresponding to the voice input status display unit 93 in Figure 2), and in response to this, mobile information terminal 2 starts the voice input function in processing step S202. When user 1 inputs voice in processing step S203, the voice is converted into a character string and then the character string is displayed in processing step S204. In processing step S205, the character string after voice input can be edited by operating the keyboard.

The character string edited by voice input and/or keyboard operation is sent to the server 3 as a search word 313 in processing step S206, and candidates are extracted by referring to the master database 31 in the server 3. In processing step S207, the candidate ingredient names are presented on the mobile information terminal 2. As is clear from the above example, the number of candidate ingredient names can be 0, 1, or 2 or more.

In processing step S208, the user 1 checks the candidate ingredient names 312 displayed on the mobile information terminal 2. If the user 1 finds a suitable candidate ingredient name 312 among the displayed candidate ingredient names 312, he or she can select that candidate ingredient name 312 and give an instruction to register it; if the user does not find a suitable candidate ingredient name 312, he or she can edit the character string to change the search word 313, or give an instruction to register the ingredient in the ingredient list 32 as it is, since it is not in the master database 31. In this way, the ingredient name 322 to be added to the ingredient list 32 is confirmed.

The information on the ingredient name for which registration has been instructed is sent to the server 3 in processing step S209, and is added to the ingredient list 32 in the server 3 in processing step S210. At this time, the ingredient details 320 can be automatically set by referring to the reference ingredient information 310 in the master database 31. The ingredient list 32 may be stored in the mobile information terminal 2.

In processing step S211, the registered ingredient list 32 and ingredient details 320 are displayed on the mobile information terminal 2. Processing step S202 moves to a voice input standby state and waits for the next voice input.

<Method 2: Allows you to register multiple ingredients at once>
FIG. 7 is a diagram showing the processing procedure of the second method. First, when voice input is started by operating the voice input status display unit 93 or inputting a wake-up word (S20), voice character input is accepted. When the end of voice input is detected by, for example, a silent period lasting for one second or more or by tapping the voice input status display unit 93 (S21, Y), it is determined that the voice input (provisional input) of one ingredient name has ended and that this is the end of one character string (ingredient name). That is, one character string is displayed as a provisional list in the character string display unit 94 (S22). In method 2, multiple ingredients can be kept in the provisional list. Therefore, the provisional list is displayed in the range of the main screen 91 where a part of the ingredient list 32 was displayed in method 1.

Furthermore, in method 2, the acceptance of voice input is resumed (S22). The following steps S23 and S25 are executed as interrupt processing. The acceptance of voice input may be resumed automatically as an operation of the continuation unit 21b, or may be resumed by operating the voice input status display unit 93.

While accepting voice input (S22), if any of the character strings (ingredient name) registered in the provisional list is selected by tapping or the like (S23, Y), the editing processing unit 24 activates the selected character string and allows character string editing to be performed (S24). Voice input is stopped while character string editing is being performed.

Furthermore, when a registration instruction is given by, for example, tapping the registration button serving as the registration instruction unit 95 (S25, Y), the master database 31 is searched for each character string registered in the provisional list as a search word 313. For each character string, if an ingredient matching the search word 313 is found, the corresponding ingredient name 312 is added to the ingredient list 32, and the reference ingredient information 310 is automatically set as the ingredient details 320 for that ingredient. If no matching ingredient is found, the ingredient details 320 are not automatically set, and the character string is added to the ingredient list 322 as the ingredient name 322. On the other hand, if a registration instruction is not given (S25, N), voice input acceptance continues (S20).

Figures 8, 9, and 10 show examples of screen transitions in each state in method 2. Figure 8 is an example of a screen transition in method 2. Figure 9 is an example of a screen transition when changing an ingredient name from the provisional list display. Figure 10 is an example of a screen transition when deleting an ingredient name from the provisional list display.

First, in FIG. 8, the screen display content transitions sequentially from screen 121 to screen 125. However, this screen is shown as an example in which only the main screen 91 and the voice input status display section 93 are displayed. First, screen 121 can be displayed in the state immediately before S20. In this screen, for example, the main screen 91 may display information to explain to user 1 how to perform the operations shown in FIGS. 9 and 10. For example, "You can edit the ingredient name by tapping the ingredient name in the temporary list" or "You can delete an ingredient by swiping the ingredient name in the temporary list." Receiving voice input is started by tapping the voice input status display section 93 in screen 121 (S20).

Screen 122 is the screen during voice input, and the voice input status display section 93 is highlighted to indicate that voice input is in progress. The character string entered by voice is displayed on the main screen 91 as the character string display section 94.

Screen 123 is a screen where voice input of one character string has been completed (S21, Y). In this example, the user speaks "carrot" and the voice is recognized as intended. After finishing speaking "carrot", voice input of one character string ends when, for example, there is silence for one second or when the voice input status display unit 93 is tapped, and the end of the character string is added. After the end is added, if there is no interruption such as editing the character string or a registration instruction, acceptance of voice input begins again (S20). Furthermore, if one or more character strings are input to the temporary list in the character string display unit 94, the registration instruction unit 95 (register button) is activated.

Screen 124 shows the case where the user similarly inputs "broccoli," "leeks," and "salmon" by combining voice input with appropriate text editing (repeating steps S20 to S25, N).

When a registration instruction is given by, for example, tapping the registration instruction unit 95 (S25, Y), the master database 31 is searched and the detailed ingredient information 320 is automatically set, as described in method 1, using each of the ingredient names (character strings) registered in the provisional list in the character string display unit 94 as search words 313.

Screen 125 shows the state after S26 is completed and transition to the list display screen of ingredient list 32. Screen 126 shows detailed display of each ingredient on the list display screen of ingredient list 32. Screen 126 displays the number of days remaining until estimated consumption date 324 (displayed as "1 day remaining") and image data 327 for an ingredient with ingredient name 322 "carrot". Also displayed superimposed on image data 327 are a vegetable icon 3251 indicating storage location 325, which is the vegetable compartment, and an "N" icon 3252 indicating that the ingredient has been newly added to ingredient list 32. In this way, various icons indicating information about the ingredient, such as storage location 325 or the fact that it has been newly added, can be displayed superimposed on image data 327 in the list display screen.

In the example of FIG. 8, the character string "sake" is a polysemous word that can be the search word 313 of the food ingredient names "sake" and "salmon." In this example, one of these words, "salmon," is automatically selected and registered in the food ingredient list 32. However, as described below, for polysemous words, an instruction from the registration instruction unit 95 may be used as a trigger, for example, to transition to a screen that allows the user to select which food ingredient in the master data 31 to select.

Figure 9 shows an example of screen transitions when editing a character string in method 2 (S23, S24). A tentative list of the character string display section 94 of the main screen 91 (tentatively displaying "carrot", "broccoli", "leeks", and "sake") and a registration instruction section 95 are displayed on screen 131. A microphone icon is displayed as the voice input status display section 93. Here, it is assumed that "carrot" in this character string display section 94 is tapped to edit it.

In the next screen 132, the character input screen 92 is displayed and the editing processing unit 24 is put into operation. In addition, the character string display unit 94 is displayed in a darkened state except for the selected character string "carrot". The character string "carrot" can be corrected to, for example, "garlic" using the keyboard on the character input screen 92. After correction, the screen 133 reflects the character string edit in the character string display unit 94.

Figure 10 shows an example of a screen transition when deleting an ingredient from the temporary list in the character string display section 94. When a predetermined operation, for example a swipe operation, is performed on the character string "carrot" in the character string display section 94 of the main screen 91, a "Delete" button is activated as shown in the next screen 142. When the "Delete" button is tapped, the screen transitions to screen 143. In other words, "carrot" is deleted from the temporary list.

Figure 11 is a sequence diagram showing a series of processing steps according to Figure 7 between user 1, mobile information terminal 2, and server 3 in the second method. In processing step S221, user 1 taps the voice input status display section 93 to indicate acceptance of voice input, and mobile information terminal 2 starts accepting voice input in processing step S222 (S20). When user 1 inputs voice in processing step S223, the voice is converted into a character string and then the character string is displayed in processing step S224, and the end of the string is determined based on a silent period or the like (S21, Y), and one ingredient is added to the tentative list (S22). The mode of waiting for voice input again is entered (S20).

In processing step S225, when an instruction to correct the character string using the keyboard is given by user 1 in processing step S224, the input character string can be corrected or deleted by operating the keyboard. The voice input and character string correction input in processing step S223 may be repeated multiple times, and processing steps S224 and S225 are executed each time accordingly. In the figure, the symbols a, b, and c represent multiple processes. The acceptance of voice input (S20) may be temporarily stopped while the character string is being edited (S23, S24).

Then, in processing step S226, user 1 issues a registration instruction (S25, Y). This registration instruction is sent to server 3 together with the provisional list in processing step S227. In processing step S228, the master database 31 in server 3 is referenced to search for each of the character strings in the provisional list as search word 313 to see if there are any matching ingredients. If there are any matching ingredients, in processing step S229, the ingredient name 312, estimated consumption period 314, etc. are obtained from master database 31, linked as ingredient details 320, and registered in ingredient list 32. When used in conjunction with the concierge function described below, an ingredient ID 311 may also be obtained.

<Linkage with Concierge Database 33>
The concierge database 33 is a database that can be created for a function independent of the food management function (food list 32 management function). For example, the concierge database 33 includes detailed information for each model of refrigerator produced by a certain manufacturer, such as what storage compartments the refrigerator has and what temperature setting patterns the refrigerator has. The concierge database 33 can be created for the concierge function that can output advice corresponding to the model of refrigerator when storing a certain food. More specifically, if a certain model of refrigerator has only a refrigerator and a chilled compartment, the concierge database 33 is a function that can advise whether it is more suitable to store each food in the refrigerator or the chilled compartment. For example, advice can be given that the chilled compartment should be used for the food name "prosciutto" and that the refrigerator should be used for the food name "milk". In addition, the location of the chilled compartment in the refrigerator of that model can be clearly indicated by using an icon that simulates the refrigerator. In addition, if the temperature setting of a certain storage compartment of the refrigerator of that model can be changed, for example, if the temperature setting of the refrigerator can be changed by the refrigeration temperature setting or the freezing temperature setting, advice can be given for each setting.

The master database 31 uses the ingredient name 312 (or ingredient ID 311) as a unique key, and includes as data items such as estimated consumption period 314 and recommended storage locations 315 at the level of storage compartments (refrigerator, freezer, vegetable compartment, etc.) of a typical refrigerator, making it suitable for general management of ingredients. On the other hand, the concierge database 33 also uses the ingredient name 332 (or ingredient ID 331) as a unique key, but includes information on the refrigerator model and settings as data items. For this reason, linking (integrating) the master database 31, which is premised on general-level information on refrigerators, with the concierge database 33, which includes detailed information on the model and settings, can contribute to the creation of convenient functions.

In this embodiment, when searching for ingredients in the concierge database 33 in the ingredient management function, the ingredient name entered or set in the ingredient management function (function that uses the ingredient list 32, etc.) is automatically entered, thereby contributing to convenience.

Figure 12 shows an example of the configuration of the master database 31 and the usage concierge database 33.

The master database 31 includes an ingredient ID 311 that uniquely corresponds to the ingredient ID 331 stored in the usage concierge database 33, an ingredient name 312 that uniquely corresponds to the ingredient ID 311, a search word 313 that corresponds to the ingredient name 312, an estimated consumption period 314 that is an indicator corresponding to the expiration date and is an estimate of consumption from the time of ingredient registration, a recommended storage location 315 that is assigned as being suitable for the ingredient indicated by the ingredient name, an ingredient category 316 that is assigned to correspond to the ingredient indicated by the ingredient name 312, standard image data 317 that shows the appearance of the ingredient indicated by the ingredient name 312, and standard image data 318 that corresponds to the ingredient category 316.

As described above, the ingredient details information 320 displayed on the mobile information terminal 2 can be automatically set with information obtained by referencing the master database 31 using the search word 313. The ingredient details information 320 may include information that is not displayed on the mobile information terminal 2, and in this embodiment includes an ingredient ID that is the same as the ingredient ID 311 in the master database 31. Then, by tapping the link button 72 on the display screen of the ingredient details information 320, the ingredient corresponding to this ingredient ID can be searched for and referenced in the concierge database 33.

Figure 13 explains the above-mentioned process. First, in processing step S31, the master database 31 is searched through the character string input of the search word 313 on the mobile information terminal 2, the ingredient name 312 "carrot" is found, and a registration instruction is issued. In response to this, in processing step S32, the master database 31 is referenced, and the information on the estimated consumption period 314, recommended storage location 315, ingredient category 316, and standard image data 317, 318 corresponding to the ingredient name 312 "carrot" is read, and these are automatically set in the ingredient list 32 as detailed ingredient information 320 (S33).

Furthermore, suppose that user 1 wishes to use the concierge function to refer to information other than the ingredient details information 320. For example, in order to obtain advice on the appropriate storage location for the ingredient name 312 "carrots" in the model of refrigerator used by user 1, the user operates the link button 72 displayed on the display screen of the ingredient details information 320 for the ingredient name 312 "carrots". This causes a search of the concierge database 33 using the ingredient ID 311 of this ingredient (there is no particular restriction as long as the link can be established, and the ingredient name 312 may also be used). As described above, the ingredient ID 311 in the master database 31 and the ingredient ID 331 in the usage concierge database 33 are unique, so that it is possible to search and refer to the corresponding ingredient. In this way, by specifying the ingredient in the ingredient management function and operating the link button 72, the concierge database 33 can be searched for this ingredient. This saves user 1 the trouble of activating the concierge function and inputting the ingredient name again.

Figure 14 is a diagram showing an example of the data structure of the concierge database 33. As described above, the ingredient ID 331 is unique to the ingredient ID 311 in the master database 31. The search word 333 is used when the user uses the concierge function, separate from the ingredient management function. Here, since the search is performed from the ingredient ID 311 in the master database 31, there is no need to worry too much about it.

The concierge database 33 has storage location/advice patterns 334 for each food ingredient, which are subdivided by refrigerator model and settings. For example, as illustrated in the storage location/advice pattern 334 in FIG. 14, a "model A" refrigerator has a so-called six-door layout, which is common to illustrations A-M shown in FIG. 15. A "model A" refrigerator is a model that can have four settings, settings 1-4. Here, settings 1-4 are different in the temperature settings of the two storage compartments (switchable compartments) shown in illustrations D and K, respectively, and the temperature settings of the storage compartments shaded in illustrations (D, K) are as follows: setting 1 = (vegetables, frozen), setting 2 = (frozen, vegetables), setting 3 = (vegetables, vegetables), setting 4 = (frozen, frozen). In addition, in all settings, the storage compartments shown in illustration A are set to a refrigerated temperature.

For example, since the ingredient name "carrot" 332 in FIG. 14 is a vegetable, the switchable compartment with the vegetable temperature setting is recommended as the storage location. If there is no switchable compartment with the vegetable temperature setting, the refrigerator compartment should be recommended. The locations of the switchable compartment for the vegetable temperature range in settings 1-4 correspond to illustrations D, K, L, and M, respectively. As the storage location/advice pattern 334, illustrations D, K, and L corresponding to the locations of the switchable compartment for the vegetable temperature range are set in settings 1, 2, and 3. On the other hand, since there is no switchable compartment for the vegetable temperature range in setting 4, illustration A is set, which indicates a storage compartment with the refrigerator temperature range.

Figure 16 shows the screen transitions when referring to the concierge database 33. Here, "tomato" is used instead of "carrot" as the ingredient to be searched for, but "carrot" would also work. First, screen 141 shows an example of the main screen 91 displayed on the mobile information terminal 2 after the reference ingredient information 310 has been reflected in processing step S33 of Figure 13. Tapping the link button 72 ("View advice on storage location") on this screen transitions to screen 142.

Screen 142 is a screen for searching for recommended storage locations for ingredients by type, as an example of the concierge function. By tapping the link button 72, the ingredient name 322, which is set by referring to the ingredient name 312 in the master database 31, is set as a search keyword in the concierge function. Here, "tomato" is searched for in the concierge database 33. For ease of explanation, the ingredient name is used here, but if the ingredient ID 311 in the master database 31 is set when registering an ingredient in the ingredient list 32, that may also be used.

The concierge function can then refer to the model information of the refrigerator used by user 1 and display the location of the recommended storage location for the selected "tomatoes" on the main screen 91 for this model of refrigerator. In this case, if the model is "Setting 1," illustration D will be displayed.

As part of the concierge function, the user can be taken to a page with further advice regarding food preservation. Screen 143 shows further advice for this model, indicating that this is the recommended storage location "when the switchable compartment is set to 'vegetables'."

Figure 17 is a sequence diagram showing the linking process procedure between the master database 31 and the concierge database 33. Following the process contents of Figure 6, in processing step S212, the user 1 taps on a food for which he/she wishes to check details from the food list 32 displayed on the mobile information terminal 2 to display the ingredient details 320. In response to this, in processing step S213, the ingredient details 320 and "View advice on storage location" (link button 72) are displayed on the mobile information terminal 2.

In processing step S214, the link button 72 is tapped. In processing step S215, the user moves to the "concierge function" screen within the app, and in processing step S216, the model of the refrigerator 4 used by user 1, which has been previously acquired, and the settings at the time of processing step S215 are acquired from the refrigerator 4. In processing step S217, the concierge database 33 is searched using the ingredient name 322 (or ingredient ID 321) corresponding to the ingredient details information 320 as a key. For matching ingredients, the storage location/advice pattern 334 in the concierge database 33 is further acquired using the information acquired in S216. In processing step S218, the acquired results are displayed on the mobile information terminal 2.

[Other features]
Next, various related functions will be described.

Presetting Storage Location 325 Options
In the above embodiment, the master database 31 has a recommended storage location 315 defined for each ingredient ID 311. However, depending on the model of the refrigerator used by the user 1, the recommended storage location 315 may not be provided. For this reason, a function may be provided for designating in advance a storage room that can be automatically set as the storage location 325 among the recommended storage locations 315.

Specifically, a function for registering the storage compartments of the refrigerator used by user 1 is added. For example, it may be registered through this function that the only storage compartments that user 1 can use are the "refrigerator" and the "freezer." In this case, for food items for which the "vegetable compartment" is set as the recommended storage location 315, for example, the "refrigerator," which has a similar temperature range, may be converted to the storage location 325 and automatically set, or the storage location 325 may not be automatically set.

<Selection and Adjustment of Automatically Inputted Detailed Ingredient Information 320>
In the above embodiment, the estimated consumption period 314, the recommended storage location 315, the food ingredient category 316, and the standard image data 317 and 318 are defined as items that are automatically set through a search of the master database 31. However, for example, there may be a user who does not wish to manage the estimated consumption period 314. For this reason, a function for selecting the items that are automatically set may be provided.

In addition, in the above embodiment, one type of period is set for each ingredient in the master database 31 as the estimated consumption period 314 (for example, five days for "pork"). However, it is conceivable that some users may be sensitive to freshness, for example, they may use a refrigerator with excellent freshness-keeping capabilities, which allows them to keep ingredients fresh for a longer period than usual. For this reason, a function may be provided that allows the period that is automatically set as the estimated consumption period 314 to be adjusted to multiple levels, such as "short," "standard," and "long."

Also, food stores may sell food items at a discount because the expiration date is approaching. Setting the normal estimated consumption period 314 for such food items may not be realistic, so a check box or the like may be provided to register the food item as a so-called "discount item." If the food item is registered in the food item list 32 with this check box checked, a shorter estimated consumption period 314 than normal may be set. Food items with this check box registered may be displayed clearly on the display screen of the food item list 32.

<Selection of which image data 317 or 318 in the master database 31 is to be reflected>
In the above embodiment, two types of standard image data 317 and 318 are prepared as image data to be automatically set as detailed ingredient information 320. For example, by preparing two types of data with completely different atmospheres, photo data and illustration data, it is possible to assume that each user will use the data after establishing their own tacit understanding, such as "photo data is for Family A to buy and use," and "illustration data is for Family B to buy and use." For this reason, the image data to be used as detailed ingredient information 320 may be selected manually each time, rather than automatically.

<Database structure for distinguishing ingredient names 312 according to preservation state>
In the above embodiment, the master database 31 has food ingredient names 312 that do not distinguish the state of the food ingredient, such as "carrot" and "broccoli." However, for example, with regard to "shrimp," there may be different storage states, such as raw shrimp that is suitable for storage in a chilled compartment, and frozen shrimp that is sold in a frozen state. For this reason, food ingredient names that distinguish the storage state, such as "shrimp (raw)" and "shrimp (frozen)," may be prepared in advance in the master database 31 or the concierge database 33 as food ingredient names 312. In this case, the recommended storage location 315 may be set to the chilled compartment or the refrigerator compartment for the former, and the freezer compartment for the latter.

<Processing when there are multiple ingredients that completely match the search word 313>
For example, when a homophoneically ambiguous character string is entered as the search word 313, two ingredient names 312 may correspond to it as an exact match. For example, the search word 313 "sake" (salmon) corresponds to two ingredient names 312, "sake" (sake) and "salmon" (salmon), and it is not possible to determine which one it is. In this case, the detailed food information 320 may be temporarily registered without being set, and then the information may be determined later.

Figure 18 is a diagram showing the process when there are multiple ingredients that perfectly match the search word 313. First, screen 151 shows the display content when a provisional list is created using method 2, for example. Screen 152 shows the display content when the ingredient names determined using the character strings in the provisional list are reflected in ingredient list 32 in response to a registration instruction. Because "sake" and "salmon" are present, which are perfect matches for search word 313 "salmon," an ingredient is not determined and it is added to ingredient list 32 as "salmon," treating it as an ingredient not in master database 31. This makes it possible to prevent incorrect ingredient names 322 and detailed ingredient information 320 from being set.

User 1 can add an appropriate ingredient by transitioning from screen 152 to the ingredient details information 320 screen for "salmon" and editing the ingredient name 322 as text, for example, to "salmon." Once "salmon" has been entered as ingredient name 322, the master database 31 is searched again, and the ingredient details information 320 is automatically set.

<Food management using refrigerator cameras>
In the above embodiment, ingredients are added to the ingredient list using voice input or text editing using a keyboard, but it is also possible to add ingredients to the ingredient list using camera images, for example by attaching a camera to the refrigerator. The camera is included in the ingredient management device shown in FIG.

First, the ingredient name and actual storage location are registered in the ingredient details information 320 and made known using the method described in the above embodiment. The actual storage location is not the recommended storage location obtained by referring to the master database 31, but the storage location in the refrigerator or other location used by user 1, which is modified by user 1 as appropriate. In this way, information that matches the actual storage can be obtained in advance.

Next, the camera captures an image of the food in the storage compartment, and this image is used to perform image recognition of the food. Publicly known methods can be used for image recognition processing. First, it is recognized whether an item is a food item or not, and the image is cut out for each food item. For example, when an image of the vegetable compartment is taken, the image is cut out for each food item, such as carrots or broccoli. At this point, image recognition can recognize the presence of food items, but does not recognize what each of the cut-out ingredients is.

Next, a process is carried out to recognize the item of each cut ingredient. At this time, it is preferable to configure the system so that information can be obtained as to which storage compartment (refrigerator, freezer, vegetable compartment, etc.) the image of which has been captured by the camera can be obtained. This can be achieved, for example, by having the user 1 input this information before capturing the image. Furthermore, the detailed ingredient information 320 for each ingredient in the ingredient list 32 is referenced to see what is stored in each storage compartment.

The accuracy of image recognition can be improved by using the storage room information of the locations where these cameras captured images and the storage location information in the ingredient list 32. That is, for example, if the ingredient list 32 contains information indicating that carrots are stored in the vegetable compartment, the image recognition function can weight the vegetable compartment so that carrots are likely to be there, and proceed with ingredient recognition.

Then, for each ingredient captured by the camera, it is determined whether it has already been registered in the ingredient list 32. For ingredients that are determined to be already registered, the image data captured by the camera can be registered as image data 327 of the ingredient details 320 for that ingredient in the ingredient list 32. In this way, image data in actual use can be used instead of standard image data 317, 318, making it possible to provide an image that is close to what user 1 remembers.

On the other hand, if an ingredient is determined to be not registered in the ingredient list 32, the process begins with adding a new ingredient. That is, the master database 31 is searched using the ingredient name obtained by image recognition as the search word 313, and detailed ingredient information 320 of the ingredient that is found is set. However, this does not apply to the recommended storage location. In addition, an image taken by the camera is set as image data 327.

At the current state of the art, it is not necessarily easy to perform image recognition with a high degree of accuracy. For this reason, image data 327 of all ingredients captured by the camera may be set and presented to user 1, and user 1 may visually determine whether or not to add them to ingredient list 32. In this case, all ingredients may be added to ingredient list 32, and user 1 may delete unnecessary ingredients, or a new list may be created and presented as a list of candidates to add.

<Processing of ingredients whose estimated consumption date 324 has passed in the ingredient list 32>
Fig. 19 is a diagram showing how to handle the display of ingredients that have passed their estimated consumption date. As shown in screen 161 of Fig. 19, multiple types of lists can be defined as ingredient lists 32. For convenience, the ingredient list 32 described in the above embodiment will be referred to as a "storage ingredient" list. This is the list shown on screen 161.

When one day has passed since the state of screen 161, the two ingredients at the top of the list on screen 161 (carrots and broccoli) will have passed the estimated consumption date 324 in their respective ingredient details information 320. When the "Stored ingredients" list is displayed again in this state, a pop-up is displayed (screen 162) indicating that ingredients whose estimated consumption date 324 has passed have been deleted from the "Stored ingredients" list. These ingredients have then been deleted from the "Stored ingredients" list (screen 163).

Figure 20 is a diagram showing how ingredients that have been deleted from the list of ingredients in storage are handled. The ingredient management device defines, for example, a list of "recently deleted ingredients" in addition to the "ingredients in storage" list. These lists are displayed as tabs on the same display screen, and it is possible to move between them with one tap. An ingredient registered in the "ingredients in storage" list can be moved to the "recently deleted ingredients" list when the estimated consumption date 324 in the ingredient details information 320 has passed (screens 171, 172). An ingredient moved to the "recently deleted ingredients" list (screen 172) can be completely deleted, for example, 48 hours after being moved.

This prevents ingredients from remaining in the "Ingredients in Storage" list, and even if user 1 neglects to update the "Ingredients in Storage" list, the list can be maintained to a certain extent in accordance with the actual stored items. On the other hand, by moving ingredients to the "Recently Deleted Ingredients" list, it is possible to discover from the "Recently Deleted Ingredients" list that there are ingredients that need to be consumed relatively soon because the estimated consumption date 324 has passed and they are still in storage.

The above-mentioned alarm function may be deactivated from the ingredient details information 320 of an ingredient that has been moved to the "recently deleted ingredients" list (screen 173). In addition, the estimated consumption date 324 may be maintained as it was in the "storage ingredients" list.

<Top screen (home screen) of food management function>
The following describes the top screen that is displayed when the food ingredient management device starts up the food ingredient management function (food ingredient management application) on the mobile information terminal 2. Fig. 21 is a diagram showing an example of the top screen displayed immediately after starting up the food ingredient management function. The top screen 181 has a function for displaying recommended ingredients, and in particular, is provided with a function for changing the recommended ingredients depending on whether or not there are ingredients in the food ingredient list 32 (the "food ingredients in storage" list) whose estimated consumption date 324 is approaching. Two screens 182 and 183 included in the top screen 181 will be described next.

If there are no ingredients in the ingredient list 32 with an approximate estimated consumption date 324 (for example, only ingredients with an estimated consumption date 324 in the next three days or more), ingredients that are in season today are displayed (recommendation display screen 182). In the recommendation display, if there are ingredients in season in the ingredient list 32, the ingredient with the fewest days remaining until the estimated consumption date 324 may be displayed.

If the ingredient list 32 contains an ingredient whose estimated consumption date is today, a display is displayed suggesting that the ingredient be consumed (reminder display, screen 183). If the ingredient list 32 contains multiple ingredients with the same estimated consumption date, they may be displayed randomly, for example. The reminder display may be a push notification sent to the mobile information terminal 2, and in response to this, a screen transition may be made to display the ingredient list 32.

<Registration reminder for estimated consumption date 324>
22 is a diagram showing screen transitions related to the reminder function for registering estimated consumption. The foodstuff management function of the foodstuff management device can be configured as an application that integrates multiple functions, such as the concierge function described above. For example, in this case or not, a menu for transitioning to the foodstuff management function can be displayed on the top screen 191, and by selecting this, transition can be made to the foodstuff management function screen (screens 192, 193).

Screen 192 shows a case where no ingredients have been registered in ingredient list 32. In this case, if an ingredient is registered and the estimated consumption date 324 is set in ingredient details information 320, a message is displayed indicating that notifications (reminders) such as recommendations for ingredients to be used can be sent accordingly.

Screen 193 shows a case where an ingredient has already been registered in ingredient list 32. When the ingredient details information 320 of a registered ingredient does not have an estimated consumption date 324 set, a message to that effect is displayed, as shown in screen 193. Whether an ingredient has an estimated consumption date 324 set can be determined by whether the ingredient does not have any information such as the number of days remaining until the estimated consumption date displayed in the list display of ingredient list 32, such as screen 193. In screen 183, for ingredients for which an estimated consumption date 324 is set, "Estimated consumption remaining D days" (D is the actual number of days remaining) is displayed.

The message indicating that there is an ingredient for which the estimated consumption date 324 has not been set may be displayed when there is an ingredient for which the estimated consumption date 324 has not been registered for several days, when the number of unregistered ingredients has exceeded a certain threshold, etc. Also, the user 1 may be able to select whether or not to display the message.

<Sorting of ingredient list 32 when user 1 owns multiple refrigerators>
The ingredient detailed information 320 may further be able to distinguish "which refrigerator" is the storage location. In the above embodiment, under the tacit understanding that the user 1 uses only one refrigerator, the storage room in a certain refrigerator, such as "refrigerator" and "vegetable room", is prepared as a storage location that can be set as the storage location 325. However, it is also possible to set two types of information in more detail as the storage location 325. Specifically, it is possible to set a storage facility such as "refrigerator 1", "refrigerator 2", or "warehouse" as the first type of information, and to set a storage location within the facility, such as "refrigerator" or "vegetable room", as the second type of information. The name of the first type of information may be definable by the user 1. The name of the second type of information may be hierarchically structured so that it changes depending on the setting of the first type of information. For example, it may be possible to define that when "refrigerator 1" is set, options such as "refrigerator" and "vegetable room" are displayed, and when "warehouse" is set, different options such as "first row from the bottom" and "second row from the bottom" are displayed.

The food management device can have filtering and sorting functions so that the food list 32 can be displayed for each storage location 325. For example, the filtering function may be a function that allows only the food ingredients stored in "Refrigerator 1" to be displayed in the food list. The filtering and sorting functions may include a function to display the food ingredients in order by storage location 325, as well as a function to filter or sort the food ingredients recently added to the food list 32 (the "food ingredients in storage" list). For recently added food ingredients, an icon indicating this may be displayed on the display screen of the food list 32 (e.g., screen 125 in FIG. 8) or the display screen of the food ingredient details 320 (e.g., screen 320 in FIG. 12). As a specific example, the "N" icon 3252 shown in FIG. 8 may be displayed.

<Variations for adding to ingredient list 32>
FIG. 23 is a diagram showing screen transitions when user 1 can select which method to use to add ingredients. When an instruction is given to activate the "ingredients management" function on top screen 201, ingredients management top screen 202 is displayed. On ingredients management top screen 192, an ingredients registration method selection button 75 is displayed along with an ingredients list 32 including ingredients that have already been registered, and when this button is operated, the screen transitions to screen 203. On screen 203, selection buttons for camera input, voice input, and text input are displayed as ingredients registration methods. By selecting any of these, any of ingredients input screens 203A, 203B, and 203C are displayed. Voice input is as in the above embodiment. In addition, text input using only the keyboard without using voice, and camera input starting from setting image data 327 are prepared.

<Detecting food turnover rate>
Although it differs depending on the user 1, it is assumed that there are foods such as eggs, dairy products, and beer that are consumed relatively daily and purchased frequently, and have a high turnover rate. If such foods can be recognized, it is expected that notifications to prevent out-of-stocks can be provided, and an example of this will be described below. Here, the above-mentioned camera is used.

For example, a method may be used to estimate ingredients that have been added to the ingredient list 32 and consumed within a short period (e.g., within 24 hours) or ingredients that have changed location or orientation in the storage location by utilizing the image recognition function of the camera described above as candidates for ingredients with a high turnover rate. Another method may be to display an image of the storage room in which ingredients are stored, captured by the camera, on the mobile information terminal 2, and have the user 1 select an ingredient for which the turnover rate is to be tracked. For ingredients with a high turnover rate or ingredients specified by the user 1, it is preferable to perform number recognition in the image recognition of the camera. In order to make it easier to perform number recognition, when the number recognition function is turned on, the ingredient management device may output a message, for example, through the display screen of the mobile information terminal 2, requesting that the ingredients to be recognized be stored in areas close to each other in the storage room. In addition, a plurality of areas for storing ingredients to be recognized may be prepared in advance on the display screen of the mobile information terminal 2 for each ingredient, so that the number recognition can be performed for each area selected by the user 1. In other words, the camera is configured to virtually divide the interior of a certain storage room into a plurality of areas in advance so that it can be recognized. For example, if there are multiple shelves in a refrigerator, each shelf can be recognized as a different area. The user can then specify that a particular food item, such as yogurt, should be placed in a specific area, such as on a particular shelf. This makes it easier for the camera to recognize the yogurt placed in that area, and therefore easier to recognize the number of yogurts and the turnover rate.

The user can specify the shelf where the yogurt is stored via the mobile information terminal 2, etc.

On the other hand, ingredients that have not been consumed for a long time or ingredients that have not changed their storage location or orientation are estimated to be candidates for ingredients with a low turnover rate. For ingredients with a low turnover rate, there is a concern that user 1 may have forgotten about their existence, so a notification may be sent at a time that is not tied to the estimated consumption date.

1: User 2: Mobile information terminal 3: Server 4: Refrigerator (storage, goods storage)
5: Cloud (Network)
31: Master database 32: Ingredient list CPU: Calculation unit Tr: Communication unit Bus: Bus 21: Voice input unit 21a: Start unit 21b: Continuation unit 22: Display output unit 23: Character conversion unit 23a: Termination unit 24: Editing unit 25: Registration unit 26: Overall processing unit 91: Main screen 92: Character input screen 93: Voice input status display unit 94: Character string display unit

Claims (18)

On one or more computers,
A start unit that starts accepting voice input;
a display output unit that displays a character string that has been recognized by speech on a display screen;
an edit processing unit that accepts editing of the character string displayed on the display screen in response to a user operation and, if edited, causes the edited character string to be displayed on the display screen;
a termination adding unit that adds a termination to a character string obtained by speech recognition and then treats the character string obtained by speech recognition as a new character string;
a registration processing unit that, in response to a user's operation, adds the character string displayed on the display screen to an ingredient list as an ingredient name ;
the display output unit displays two or more character strings,
the editing processing unit is capable of editing each character string displayed by the display output unit,
The registration processing unit is an ingredient management system that adds each character string displayed by the display output unit as an ingredient name to an ingredient list,
A master database that stores ingredient names and ingredient detail information corresponding to the ingredient names can be referenced;
During the operation of the registration processing unit, the master database is searched for an ingredient name corresponding to each character string displayed by the display output unit, and for a character string having a corresponding ingredient name, the ingredient name and the corresponding ingredient detailed information can be added as a pair to an ingredient list;
In this ingredient management system, when the master database is searched using the character string displayed by the display output unit as a search word and multiple perfectly matching candidate ingredients are obtained, ingredient detail information is not set for the ingredient name even if the registration processing unit is operated . On one or more computers,
A start unit that starts accepting voice input;
a display output unit that displays a character string that has been recognized by speech on a display screen;
an edit processing unit that accepts editing of the character string displayed on the display screen in response to a user operation and, if edited, causes the edited character string to be displayed on the display screen;
a termination adding unit that adds a termination to a character string obtained by speech recognition and then treats the character string obtained by speech recognition as a new character string;
a registration processing unit that, in response to a user's operation, adds the character string displayed on the display screen to an ingredient list as an ingredient name ;
The display output unit is a food ingredient management system that displays one character string,
automatically operating the initiation unit after the operation of the registration processing unit;
A master database that stores ingredient names and ingredient detail information corresponding to the ingredient names can be referenced;
A search is made for an ingredient name corresponding to the character string displayed by the display output unit in the master database, and if there is a corresponding ingredient name, the display output unit is caused to display it as a candidate ingredient;
When the selection of the candidate ingredient displayed on the display screen is received from the user, the registration processing unit adds the ingredient name of the candidate ingredient and the corresponding ingredient detail information as a pair to an ingredient list;
In this ingredient management system, when the master database is searched using the character string displayed by the display output unit as a search word and multiple perfectly matching candidate ingredients are obtained, detailed ingredient information is not set for the ingredient name even if the registration processing unit is operated . On one or more computers,
A start unit that starts accepting voice input;
a display output unit that displays a character string that has been recognized by speech on a display screen;
an edit processing unit that accepts editing of the character string displayed on the display screen in response to a user operation and, if edited, causes the edited character string to be displayed on the display screen;
a termination adding unit that adds a termination to a character string obtained by speech recognition and then treats the character string obtained by speech recognition as a new character string;
a registration processing unit that, in response to a user's operation, adds the character string displayed on the display screen to an ingredient list as an ingredient name ;
The food ingredient management system outputs a message urging the user to set an estimated consumption date depending on the presence of food ingredients included in the food ingredient list for which time has passed without an estimated consumption date being set, or the number of food ingredients for which an estimated consumption date has not been set . The foodstuff management system according to claim 3 ,
the display output unit displays two or more character strings,
the editing processing unit is capable of editing each character string displayed by the display output unit,
The registration processing unit adds each of the character strings displayed by the display output unit as an ingredient name to an ingredient list. The foodstuff management system according to claim 4 ,
A master database that stores ingredient names and ingredient detail information corresponding to the ingredient names can be referenced;
When the registration processing unit is operating, the ingredient management system searches the master database for ingredient names corresponding to each character string displayed by the display output unit, and for character strings that have a corresponding ingredient name, adds the ingredient name and corresponding ingredient details information as a pair to an ingredient list. The foodstuff management system according to claim 3 ,
the display output unit displays one character string,
The food ingredient management system automatically operates the initiation unit after the registration processing unit operates. The foodstuff management system according to claim 6 ,
A master database that stores ingredient names and ingredient detail information corresponding to the ingredient names can be referenced;
A search is made for an ingredient name corresponding to the character string displayed by the display output unit in the master database, and if there is a corresponding ingredient name, the display output unit is caused to display it as a candidate ingredient;
When a selection of a candidate ingredient displayed on the display screen is accepted from a user, the registration processing unit adds the ingredient name of the candidate ingredient and the corresponding detailed ingredient information as a pair to an ingredient list. The foodstuff management system according to any one of claims 1, 2, 5 and 7 ,
A concierge database is provided separately from the master database, and can refer to ingredients corresponding to the ingredients in the master database by ingredient name or ingredient ID.
By specifying a certain ingredient in the ingredient list, the ingredient name or ingredient ID of the specified ingredient can be automatically input as a search key in the concierge database. The foodstuff management system according to any one of claims 1, 2, 5 and 7 ,
The detailed information of ingredients includes a storage location of the storage facility,
A food ingredient management system in which a user can specify in advance the type of storage room that can be set as the storage location. The foodstuff management system according to any one of claims 1, 2, 5 and 7 ,
The detailed information of ingredients includes an estimated consumption period of the ingredients,
This foodstuff management system allows a user to specify in advance the length of the period to be set as the estimated consumption period. The foodstuff management system according to any one of claims 1, 2, 5 and 7 ,
The detailed ingredient information includes two or more types of standard image data of ingredients,
The ingredient management system allows a user to specify the type of ingredient to be set as the standard image data when adding each ingredient to an ingredient list. The foodstuff management system according to any one of claims 1, 2, 5 and 7 ,
In this food ingredient management system, two or more food ingredient names indicating the same food ingredient but different storage conditions are prepared in the master database. The foodstuff management system according to any one of claims 1, 2, 5 and 7 ,
The food management system is equipped with a camera,
An image recognition process is performed on the image captured by the camera to cut out each of the ingredients included in the image,
For ingredients whose names are recognized as a result of image recognition,
An image of the ingredient cut out from the image is set as the ingredient detail information of the ingredient,
At least a part of the ingredient detail information corresponding to the ingredient name is acquired from the master database, and set as the ingredient detail information of the ingredient;
The ingredient is added to the ingredient list. The foodstuff management system according to any one of claims 1, 2, 5 and 7 ,
The food management system is equipped with a camera,
An image recognition process is performed on the image captured by the camera to cut out each of the ingredients included in the image,
The ingredient management system displays ingredient images cut out for each ingredient as a result of image recognition, and asks the user whether or not each ingredient needs to be added to the ingredient list. The foodstuff management system according to any one of claims 1 to 7 ,
The ingredient list includes a list of ingredients currently being stored and a list of ingredients that have been deleted,
In this ingredient management system, ingredients included in the stored ingredient list that have passed their estimated consumption date are moved to the deleted ingredient list. The foodstuff management system according to any one of claims 1 to 7 ,
In the food ingredient management system, if the food ingredient list includes an ingredient whose estimated consumption date is approaching, a reminder is displayed to recommend consumption of the ingredient, and if not, a recommendation display is displayed to display ingredients that are in season. The foodstuff management system according to any one of claims 1 to 7 ,
To provide an ingredient management system that allows setting two or more storage locations for one ingredient as ingredient detail information. The foodstuff management system according to any one of claims 1 to 7 ,
The food ingredient management system includes a camera that is set to virtually divide a storage room into a plurality of areas and recognize the divided areas,
The food ingredient management system receives a user's instruction to designate one of the plurality of areas to store a specific ingredient, and performs count recognition and/or tracking of turnover rate of the specific ingredient stored in the area.

Images