JP7478971B2 - PROGRAM, TERMINAL DEVICE, AND SERVICE PROVIDING SYSTEM - Google Patents

Description

This disclosure relates to a program, a terminal device, and a service providing system.

Patent Document 1 discloses a technology that acquires food information from code information attached to food, acquires microwave oven heating information corresponding to the acquired food information from a server device, and transmits the acquired heating information to a microwave oven.

JP 2006-275489 A

The purpose of this disclosure is to improve user convenience based on the code information attached to food products.

The program of the present disclosure causes a computer to execute a detection step of detecting code information contained in an image captured by a photographing unit, a first determination step of determining an intended use of the code information based on an operation received by an operation unit, a transmission step of sending a request for information corresponding to the determined intended use of the code information to a server device, and a display step of displaying information received from the server device, and in the transmission step, if the intended use of the code information determined by the first determination step is cooking to heat food, a request for heating information related to cooking is sent to the server device .

The terminal device of the present disclosure comprises a display unit, a communication unit, a photographing unit, a code detection unit that detects code information contained in an image photographed by the photographing unit, an operation unit that accepts operations, and a control unit that determines a use of the code information based on the operation accepted by the operation unit and sends a request to obtain information corresponding to the determined use to a server device via the communication unit, and the control unit displays the information received from the server device on the display unit , and if the determined use of the code information is cooking to heat food, sends a request to obtain heating information related to cooking to the server device.

The service providing system of the present disclosure comprises a terminal device having a display unit, a communication unit, a photographing unit, a code detection unit that detects code information contained in an image photographed by the photographing unit, an operation unit that accepts operations, and a control unit that determines a use of the code information based on the operation accepted by the operation unit and sends an acquisition request for information corresponding to the determined use to a server device via the communication unit, and the server device that transmits information corresponding to the acquisition request to the terminal device, wherein the control unit displays the information received from the server device on the display unit, and if the determined use of the code information is cooking to heat food, sends a request to acquire heating information related to cooking to the server device.

The program, terminal device, and service providing system disclosed herein can determine the use of the code information based on the received operation, and can obtain and display information corresponding to the determined use from the server device. Therefore, even when the same code information is read, information corresponding to the operation can be displayed on the display unit, improving user convenience.

System configuration diagram of the service provision system A block diagram showing the configuration of a terminal device, a cooking appliance, and a server device. Diagram showing the structure of the user database FIG. 13 is a diagram showing an example of a home screen. FIG. 13 is a diagram showing an example of a processing selection screen. A diagram showing an example of a scan screen FIG. 13 is a diagram showing an example of a selection screen. A diagram showing an example of a scan screen FIG. 13 is a diagram showing an example of a confirmation screen. A diagram showing an example of a scan screen A diagram showing an example of a scan screen FIG. 13 is a diagram showing an example of a selection screen. FIG. 13 is a diagram showing an example of a confirmation screen. A diagram showing an example of a scan screen A diagram showing an example of a search result screen Flowchart showing the operation of the terminal device Flowchart showing the operation of the terminal device Flowchart showing the operation of the terminal device Flowchart showing the operation of the server device

Hereinafter, the embodiments will be described in detail with reference to the drawings. However, in some cases, more detailed explanations than necessary may be omitted. For example, detailed explanations of already well-known matters or duplicate explanations of substantially the same configurations may be omitted.
It should be noted that the accompanying drawings and the following description are provided to enable those skilled in the art to fully understand the present disclosure, and are not intended to limit the subject matter described in the claims.

(Embodiment 1)
Hereinafter, the first embodiment will be described with reference to FIGS.
[1-1. Configuration]
FIG. 1 shows the system configuration of a service providing system 1 .
The service providing system 1 has a configuration in which a terminal device 100, a cooking appliance 200, and a server device 300 are connected via a network 10. The network 10 may be a wired network, or may include a wireless network in part. The network 10 may be configured to include a wide area network such as the Internet, a mobile communication network, or a WAN (Wide Area Network), or may be configured as a private network such as an intranet or a LAN (Local Area Network). The terminal device 100 and the cooking appliance 200 may be connected to the network 10 in a wired manner, or may be connected wirelessly such as Wi-Fi (registered trademark).

The terminal device 100 is a portable information processing device, and is a terminal carried by a user who receives the provision of the service. The terminal device 100 may be, for example, a smartphone, a tablet PC (Personal Computer), a notebook PC, a desktop PC, a mobile phone, a PDA (Personal Digital Assistant), or the like. In this embodiment, the terminal device 100 will be described as a smartphone.

A service application program (hereinafter simply referred to as service app 143) is installed in the terminal device 100. This service app 143 may be downloaded from the server device 300 by the user operating the terminal device 100, or may be downloaded from a server device other than the server device 300.

The terminal device 100 executes the service app 143 to provide services to the user. Services provided by executing the service app 143 include, for example, services for setting heating information such as the rated high frequency output and heating time in the cooker 200 when cooking food such as frozen food using the cooker 200, for example, for managing the inventory of food and ingredients purchased by the user, and for searching for recipes that include ingredients selected by the user.

Cooking appliance 200 is a cooking appliance that heats and cooks food. In this embodiment, a case where cooking appliance 200 is an oven range will be described, but cooking appliance 200 may be, for example, a rice cooker or an electric pressure cooker.
Cooking appliance 200 is connected to server device 300 via network 10. Cooking appliance 200 sets cooking settings for heat cooking based on data received from server device 300. When a cooking start button is pressed by a user with cooking settings set, cooking appliance 200 starts heat cooking according to the set cooking settings.
In this embodiment, the terminal device 100 and the cooking appliance 200 do not communicate directly with each other. However, the terminal device 100 and the cooking appliance 200 may communicate directly with each other via short-range wireless communication such as Bluetooth (registered trademark) to transmit and receive data.

The server device 300 provides information to the terminal device 100 owned by a registered user. The information provided by the server device 300 includes, for example, information on the heating information of the cooker 200, expiration dates of food, storage methods and expiration dates of ingredients, recipes, and the like.
Furthermore, when server device 300 receives heating information from terminal device 100, it transmits the received heating information to cooking appliance 200 associated with terminal device 100 that received the information. In other words, server device 300 transmits heating information selected by the user of terminal device 100 to cooking appliance 200 used by the user, and performs cooking settings for cooking appliance 200.

2 is a block diagram showing the configurations of the terminal device 100, the cooking appliance 200, and the server device 300. First, the terminal device 100 will be described.
The terminal device 100 includes a terminal communication I/F 111, a camera 113, a display unit 115, and a terminal control unit 130.

The terminal communication I/F 111 is a communication interface equipped with a communication module, and performs data communication with the server device 300 via the network 10. The communication method of the terminal communication I/F 111 may be, for example, a wireless communication method such as 4G (Generation), 5G, or Wi-Fi. The terminal communication I/F 111 corresponds to an example of a communication unit.

The camera 113 includes a lens and an imaging element. The imaging element is, for example, an image sensor such as a CCD (Charged Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor). The camera 113 generates a captured image by capturing an image, and outputs the generated captured image to the terminal control unit 130. The camera 113 corresponds to an example of a capture unit.

The display unit 115 includes a touch panel 117 that includes a display panel such as a liquid crystal or organic EL (Electronic Luminescent) panel, and a touch sensor that detects touch operations. The touch panel 117 corresponds to an example of an operation unit. The touch sensor outputs coordinate information to the terminal control unit 130 that indicates the position on the display panel that is touched by a user's finger or a pointer such as a pen. This coordinate information is the coordinate in a preset coordinate system of the display panel.

The terminal control unit 130 is a computer equipped with a terminal memory unit 140 and a terminal processor 150.

The terminal storage unit 140 includes, for example, a non-volatile storage device such as a flash memory or an SD card, and a volatile storage device such as a RAM (Random access memory).
The terminal storage unit 140 stores a control program 141 such as an OS (Operating System), a service application 143, a user ID (identifier) 145, and various other setting data. The user ID 145 is information set by the user of the terminal device 100, and is identification information for identifying a user of the service providing system 1. The terminal storage unit 140 also stores inventory management data for food and ingredients purchased by the user. The inventory management data includes data such as the type (name), quantity, expiration date, best before date, and storage method of the food and ingredients.

The terminal processor 150 is configured with a microcomputer such as a CPU (Central Processing Unit), an MCU (Micro Controller Unit) equipped with a CPU, or an MPU (Micro Processor Unit). The terminal processor 150 may also be configured as part of a SoC (System-on-a-Chip) that includes the terminal communication I/F 111, a camera, various sensors, and memory.

The terminal control unit 130 has, as its functional configuration, a code detection unit 151, an image recognition unit 153, a data acquisition unit 155, and a display control unit 157. These are functions that are realized when the terminal processor 150 executes the control program 141 and the service app 143 stored in the terminal storage unit 140.

The code detection unit 151 detects code information from the image captured by the camera 113. The code information acquired by the code detection unit 151 includes, for example, one-dimensional codes such as JAN codes and two-dimensional codes such as QR codes (registered trademark). The code detection unit 151 detects code information such as one-dimensional codes and two-dimensional codes from the captured image and acquires identification data for identifying food contained in the detected code information. The operating mode of the terminal control unit 130 when the code detection unit 151 is operating is called the code recognition mode.

The image recognition unit 153 identifies an object captured in an image captured by the camera 113. For example, the image recognition unit 153 extracts feature points such as shape, color, size, etc. contained in the captured image, and compares the extracted feature points with pre-stored feature points of ingredients to determine the type of ingredient. The operating mode of the terminal control unit 130 when the image recognition unit 153 is operating is called the image recognition mode.

The data acquisition unit 155 controls the terminal communication I/F 111 to transmit a data acquisition request to the server device 300 and acquire the data from the server device 300 .
When code detection unit 151 detects code information including identification data for identifying a food from the image captured by camera 113, and the operation detected by touch panel 117 is an operation for selecting a process for transmitting heating information to cooking appliance 200, data acquisition unit 155 determines that the use of the code information is acquisition of heating information. In this case, data acquisition unit 155 transmits a request for acquiring heating information including the food identification data and user ID 145 to server device 300, and acquires data including the heating information.

Furthermore, when the code detection unit 151 detects code information including food identification data from the image captured by the camera 113, and the operation detected by the touch panel 117 is an operation to select inventory management, the data acquisition unit 155 determines that the use of the code information is inventory management. In this case, the data acquisition unit 155 transmits a request to the server device 300 to acquire information necessary for inventory management, including the food identification data and the user ID 145. Information necessary for inventory management includes the expiration date of the food, storage method, etc. Hereinafter, information necessary for food inventory management is referred to as food management information.

Furthermore, when the image recognition unit 153 has determined the type of ingredient and the operation detected by the touch panel 117 is an operation to select inventory management, the data acquisition unit 155 sends to the server device 300 a request to acquire data necessary for inventory management, including information indicating the determined type of ingredient and the user ID 145. Information necessary for inventory management includes the expiration date and storage method of the ingredient. Hereinafter, information necessary for inventory management of ingredients is referred to as ingredient management information.

In addition, when the image recognition unit 153 determines the type of ingredient and the operation detected by the touch panel 117 is an operation to select a recipe search, the data acquisition unit 155 transmits a request to acquire recipe data including information indicating the determined type of ingredient and the user ID 145 to the server device 300.

The display control unit 157 controls the display unit 115 to display a display screen on the touch panel 117. The display control unit 157 generates display data and outputs the generated display data to the display unit 115. The display unit 115 displays a display screen on the touch panel 117 based on the input display data. For example, the display control unit 157 displays on the touch panel 117 a package image of the food, heating information, the expiration date of the food, storage methods, recipes, etc., received from the server device 300.

The cooking appliance 200 includes a cooking appliance communication I/F 201, an operation unit 203, a display unit 205, a cooking control unit 210, and a heat cooking unit 220. The cooking appliance communication I/F 201 is a communication interface that includes a communication module. The cooking appliance communication I/F 201 performs data communication with the server device 300, for example, via wireless communication such as Wi-Fi.

The operation unit 203 includes, for example, a cooking start button and setting buttons for setting the heating time, rated high frequency output, etc. When the operation unit 203 receives an operation of the cooking start button or a setting button, it outputs an operation signal corresponding to the button that received the operation to the cooking control unit 210.

The display unit 205 has a display panel such as a liquid crystal panel, and displays characters and images on the display panel according to the control of the cooking control unit 210. For example, the display panel displays an image of the food to be cooked, heating information such as the heating time, and rated high-frequency output.

The cooking control unit 210 controls the heat cooking unit 220 in accordance with operations accepted by the operation unit 203 and heating information received from the server device 300, and performs heat cooking.
In addition, the cooking control unit 210 causes the display unit 205 to display the operation content accepted by the operation unit 203 and the heating information received from the server device 300.

The cooking unit 220 cooks the food contained in the heating chamber (not shown) according to the cooking method and cooking time set by the cooking control unit 210. Cooking methods include microwave cooking, oven cooking, steam cooking, grill cooking, etc.

Next, the configuration of the server device 300 will be described.
The server device 300 includes a server communication I/F 301 and a server control unit 310 .
The server communication I/F 301 includes a network adapter such as a network interface card (NIC), and performs data communication with the terminal device 100 and the cooking appliance 200 via the network 10 .

The server control unit 310 includes a server storage unit 320 and a server processor 330 .
The server storage unit 320 is configured with, for example, a non-volatile storage device such as a ROM or a flash memory, and an auxiliary storage device such as a hard disk drive or a solid state drive (SSD). The server storage unit 320 stores a control program 321 and a database 323.

The data registered in database 323 includes heating information of the registered foods. The foods registered in database 323 are foods provided by companies participating in service provision system 1. The heating information includes information on the rated high-frequency output and heating time. For example, when the heating time varies depending on the number of foods to be cooked simultaneously, such as grilled rice balls or fried chicken, the information on the rated high-frequency output and heating time is set for each number of foods to be cooked simultaneously.
The data registered in the database 323 also includes package images of the registered food products, expiration dates, storage methods, and other data.
The data registered in the database 323 also includes expiration dates and storage methods for each food ingredient such as vegetables, meat, and fish.
The data registered in the database 323 also includes recipe data for cooking.

Figure 3 is a diagram showing the configuration of user database 323a included in database 323. User database 323a is a database that registers information about users registered in service providing system 1. In user database 323a, the type and model information of the cooking appliance 200 used by the user and the communication address of the cooking appliance 200 are registered in association with a user ID and password, which are identification information of the user.

The server processor 330 is composed of, for example, a CPU, MCU, and MPU. The server control unit 310 executes various processes by the server processor 330 executing the control program 321 stored in the server storage unit 320.

For example, when the server control unit 310 receives a request to obtain heating information from the terminal device 100, it reads out from the database 323 a food package image and heating information corresponding to the identification data included in the received request, and transmits the read data to the terminal device 100. In addition, when the server control unit 310 receives from the terminal device 100 a request to obtain food management information including food identification data and a user ID 145, it reads out from the database 323 a food package image and data such as the food's expiration date and storage method corresponding to the identification data included in the received request, and transmits the read data to the terminal device 100.

When the server control unit 310 receives a request from the terminal device 100 to obtain ingredient management information including information indicating the type of ingredient and the user ID 145, the server control unit 310 reads data such as the expiration date and storage method of the corresponding ingredient from the database 323 and transmits the read data to the terminal device 100. When the server control unit 310 receives a request from the terminal device 100 to obtain recipe data including information indicating the type of ingredient and the user ID 145, the server control unit 310 reads recipe data including the ingredients indicated by the received information from the database 323 and transmits the read data to the terminal device 100.

When the server control unit 310 receives a request to obtain heating information from the terminal device 100, it obtains the communication address of the cooking appliance 200 associated with the user ID 145 of the user who owns the terminal device 100, and transmits the heating information to the obtained communication address.

FIG. 4 is a diagram showing an example of the home screen 50. As shown in FIG.
When the service application 143 is selected by a touch operation by the user, the terminal control unit 130 launches the service application 143 and displays the home screen 50 shown in FIG.
On the home screen 50, a menu icon 51, a scan icon 53, notification screens 55 and 57, etc. are displayed.
The menu icon 51 is an icon that, when selected, displays a list of menus.
The scan icon 53 is an icon that scans the ingredients or code information attached to the food using the image captured by the camera 113, and executes processing based on the scan results. Hereinafter, the process of recognizing the ingredients or code information attached to the food using the image captured by the camera 113 will be referred to as “scanning.”
The notification screens 55 and 57 display information such as notices and recommended recipes. Although Fig. 4 shows an example in which two notification screens 55 and 57 are displayed, the number of notification screens 55 and 57 displayed on the home screen 50 is arbitrary.

FIG. 5 is a diagram showing an example of a process selection screen 510 that is displayed on the touch panel 117 when the scan icon 53 is selected.
The process selection screen 510 is a screen that displays a list of processes that are executed using scanning. A first process display 511, a second process display 513, and a third process display 515 are displayed in a selectable manner on the process selection screen 510. The first process display 511 is selected by the user touching the area of the touch panel 117 where the first process display 511 is displayed. The same applies to the second process display 513 and the third process display 515.

The first process is displayed in first process display 511. The first process is a process of transmitting heating information to cooking appliance 200.
The second process is displayed in the second process display 513. The second process is a process for managing the inventory of the scanned food or ingredients.
The third process is displayed in the third process display 515. The third process is a process for searching for recipes that include the scanned ingredient.

FIG. 6 is a diagram showing an example of a scan screen 520 displayed on the touch panel 117 when the first process display 511 is selected.
On the scan screen 520 shown in FIG. 6, a switching display 521, a guide display 523, and a range image 525 are displayed.
The switching display 521 is a display for switching the operation mode of the terminal control unit 130 between a code recognition mode for recognizing code information such as one-dimensional codes and two-dimensional codes, and an image recognition mode for extracting feature points from a captured image and recognizing ingredients based on the extracted feature points. The switching display 521 displays code recognition 521A and image recognition 521B. When the code recognition 521A is selected by a user's operation, the terminal control unit 130 changes the operation mode to the code recognition mode. Also, when the image recognition 521B is selected by a user's operation, the terminal control unit 130 changes the operation mode to the image recognition mode. FIG. 6 shows a state in which the code recognition 521A, indicated by diagonal hatching, is selected. The display of the scan screen 520 displayed when the code recognition 521A is selected is called the code recognition display.

Guidance display 523 displays a display informing the user whether code recognition 521A or image recognition 521B included in switching display 521 has been selected. For example, when code recognition 321A is selected, a message such as "Automatically recognizes one-dimensional and two-dimensional codes" is displayed. When image recognition 521B is selected, a message such as "Analyzes images to recognize ingredients" is displayed.

In range image 525, an image showing the recognition range of code recognition 521A or image recognition 521B is displayed. In other words, code recognition or image recognition is performed on images captured by camera 113 that are within the range shown in range image 525.

FIG. 7 is a diagram showing an example of a selection screen 530 that is displayed on the touch panel 117 when code information attached to a food item is recognized on the scan screen 520 shown in FIG.
When the first process display 511 is selected, the terminal control unit 130 determines that the purpose of the code information is to obtain heating information. The terminal control unit 130 transmits the obtained data to the server device 300 and obtains, from the server device 300, the heating information of the food corresponding to the code information.
Selection screen 530 displays a food package display field 531, a list display field 533 displaying a list of heating information that can be set in cooking appliance 200, and a send button 535.
The food package displayed in the food image display area 331 is an image that the terminal device 100 has obtained from the server device 300. When the terminal control unit 130 reads the code information, it transmits the read code information to the server device 300. The server device 300 transmits the food package image corresponding to the code information received from the terminal device 100 and heating information to the corresponding terminal device 100.

In the list display section 533, the rated high frequency output 5331 and the heating time 5333 are displayed as heating information. In the case of a food item whose cooking time varies depending on the number of pieces, the heating time 5333 displays the heating time for each number of pieces of food item.

The send button 535 is a button that sends information on the rated high frequency output 5331 and heating time 5333 selected by the user to the cooking appliance 200. When the send button 535 is pressed, the terminal control unit 130 sends the information on the rated high frequency output 5331 and heating time 5333 to the server device 300. The server device 300 sends the information on the rated high frequency output 5331 and heating time 5333 received from the terminal device 100 to the corresponding cooking appliance.

Figure 8 shows an example of a scan screen 540 displayed on the touch panel 117 when the second process display 513 shown in Figure 5 is selected, and shows a scan screen 540 displaying code recognition. The scan screen 540 shown in Figure 8 is the same as the scan screen 520 shown in Figure 6, so a detailed description of the scan screen 540 will be omitted.

FIG. 9 is a diagram showing an example of a confirmation screen 550 that is displayed on the touch panel 117 when code information attached to a food item is recognized on the scan screen 540 shown in FIG.
When the second process display 513 is selected, the terminal control unit 130 determines that the use of the code information is inventory management of the scanned food. The terminal control unit 130 transmits a request for obtaining food management information, including the food identification data and the user ID 145, to the server device 300, and obtains the food management information of the food from the server device 300.
The confirmation screen 550 displays a food package display field 551, a food information display field 553, and a registration button 555. The food package displayed in the food image display field 331 is an image that the terminal device 100 has acquired from the server device 300.
In the food information display section 553, an expiration date 553A and a storage method 553B are displayed.
The register button 555 is a button for registering in inventory management. When the register button 555 is pressed, the terminal control unit 130 registers the food displayed in the display field 551 in inventory management. That is, the quantity data that manages the number of the food displayed in the display field 551 is incremented by one.

FIG. 10 is a diagram showing an example of the scan screen 540 for displaying code recognition, which is displayed after the registration button 555 is pressed on the confirmation screen 550 shown in FIG.
When the registration button 555 is pressed, the terminal control unit 130 again displays the scan screen 540 shown in Fig. 10 so that the food can be scanned continuously. The scan screen 540 shown in Fig. 10 displays, as a guide display 543, that the food has been registered in inventory management and that the food can be scanned continuously.
Furthermore, the scan screen 540 shown in FIG. 10 displays an end button 547 for ending the scanning of the food item.

Fig. 11 is a diagram showing an example of the scan screen 540 displayed on the touch panel 117 when the second process display 513 shown in Fig. 5 is selected, and shows the case where image recognition 541B is selected. Fig. 11 shows a state where image recognition 521B, indicated by diagonal hatching, is selected. The display of the scan screen 520 displayed when image recognition 521B is selected is called image recognition display.
When image recognition 541B is selected, guidance display 543 displayed on scan screen 540 displays guidance such as "Ingredients will be detected from the image."
Furthermore, when image recognition 541B is selected, a scan button 549 for starting scanning is displayed. The user moves the position of terminal device 100 so that the image of the ingredient to be recognized, among the images captured by camera 113 and displayed on touch panel 117, falls within the range indicated by range image 545. When the image of the ingredient to be recognized is displayed within the range, the user presses scan button 549.

When the scan button 549 is pressed, the terminal control unit 130 extracts feature points from the captured image of the range indicated by the range image 545, and compares the extracted feature points with pre-stored feature points of ingredients to determine the type of ingredient.

FIG. 12 is a diagram showing an example of a selection screen 560 displayed on touch panel 117 when image recognition is executed on scan screen 540 shown in FIG.
Selection screen 560 displays an ingredient selection field 561 and a selection field 563. Ingredient selection field 561 displays the names of ingredients that are candidates detected by image recognition, and check boxes. The check boxes are displayed in association with the displayed ingredient names. When the ingredient names of ingredients that have been subjected to image recognition are displayed in ingredient selection field 561, the user checks the corresponding check boxes.

In the selection column 563, three buttons 563 are displayed.
The first is button 563A, which is pressed when registering the ingredient selected by the check box in inventory management. The second is button 563B, which is pressed when there is no corresponding ingredient. The third is button 563C, which is pressed when performing image recognition again. When button 563B is pressed, the terminal control unit 130 causes the touch panel 117 to display, for example, major categories such as vegetables, meat, fish, and fruit, and minor categories such as tomato, lettuce, and komatsuna. The user checks the relevant items in the major and minor categories.

FIG. 13 is a diagram showing an example of a confirmation screen 570 that is displayed when the check box for tomato displayed in the ingredient selection column 561 on the selection screen 560 shown in FIG. 12 is checked and the process of registering in inventory management is selected.
When the second process display 513 is selected, the terminal control unit 130 determines that the use of the code information is inventory management of the scanned ingredient. The terminal control unit 130 sends a request to the server device 300 to obtain ingredient management information including information indicating the ingredient type and the user ID 145, and obtains the ingredient management information of the ingredient from the server device 300.
Confirmation screen 570 displays a display field 571 for the ingredient name to be registered in inventory management, an ingredient information display field 571, and a registration button 573. In ingredient information display field 571, expiration date 571A and storage method 571B are displayed.
The register button 573 is a button for registering in inventory management. When the register button 573 is pressed, the terminal control unit 130 registers in inventory management the ingredients displayed in the display field 551. In other words, the quantity data that manages the number of ingredients displayed in the display field 551 is incremented by one.

FIG. 14 is a diagram showing an example of a scan screen 580 displayed on the touch panel 117 when the third process display 515 is selected.
When the third process display 515 is selected, a scan screen 580 for image recognition display is displayed on the touch panel 117 .
The scan screen 580 displays a switching display 581, a guide display 583, a range image 585, and a scan button 587. When the third process display 515 is selected, the scan screen 580 is changed to an image recognition display. The scan screen 580 is the same screen as the scan screen 540 shown in FIG. 11.

FIG. 15 is a diagram showing an example of a search result screen 590.
When an ingredient is detected by image recognition on the scan screen 580 shown in Fig. 14, the terminal control unit 130 displays a search result screen 590 shown in Fig. 15. The search result screen 590 shown in Fig. 15 shows the search result screen 590 when an onion is detected by image recognition. When an onion is detected by image recognition, the terminal control unit 130 transmits to the server device 300 a request to acquire recipes using the detected onion. The server device 300 searches the database 323 to read out recipe data for recipes using onions, and transmits the read recipe data to the corresponding terminal device 100.

[1-2. motion]
16 to 18 are flowcharts showing the operation of the terminal device 100.
The operation of the terminal device 100 will be described with reference to the flowcharts of FIGS.
When the service application 143 is selected by a user operation, the terminal control unit 130 executes the service application 143 and displays the home screen 50 on the touch panel 117. After that, the terminal control unit 130 If the scan icon 53 has not been operated (step S1/NO), the terminal control unit 130 waits to start processing until the scan icon 53 is operated.

When the scan icon 53 is operated (step S1/YES), the terminal control unit 130 causes the process selection screen 510 shown in FIG. 5 to be displayed on the touch panel 117 (step S2). Next, the terminal control unit 130 determines whether the first process, transmission of heating information, has been selected (step S3). The terminal control unit 130 determines whether the first process display 511 has been selected, and determines whether the first process has been selected.

When the first process is selected (step S3/YES), the terminal control unit 130 changes the display mode of the scan screen 520 shown in FIG. 6 to a code recognition display (step S4) and changes the operation mode to the code recognition mode. Then, the terminal control unit 130 determines whether or not a one-dimensional code or a two-dimensional code has been detected from the image captured by the camera 113 (step S5). When the terminal control unit 130 cannot detect a one-dimensional code or a two-dimensional code (step S5/NO), it repeats the determination of step S5 until a one-dimensional code or a two-dimensional code is detected. Step S5 corresponds to the detection procedure.

When the one-dimensional code or two-dimensional code is detected (step S5/YES), the terminal control unit 130 acquires the data contained in the detected one-dimensional code or two-dimensional code. When the first process display 511 is selected in the judgment of step S3, the terminal control unit 130 judges that the use of the code information is to acquire heating information. The terminal control unit 130 transmits the acquired data to the server device 300 and acquires the heating information of the food indicated by the identification data from the server device 300 (step S6). When the terminal control unit 130 acquires the heating information from the server device 300, it displays the acquired heating information on the selection screen 530 shown in FIG. 7 and displays a list of heating information that can be set on the cooker 200 (step S7). Steps S5 and S6 correspond to the first judgment procedure and the transmission procedure. Also, step S7 corresponds to the display procedure.

Next, the terminal control unit 130 determines whether or not one of the heating information items has been selected (step S8). If the terminal control unit 130 has not received an operation to select heating information (step S8/NO), the terminal control unit 130 waits to start processing until heating information is selected.

When the terminal control unit 130 receives an operation to select heating information (step S8/YES), it determines whether the send button 535 shown in FIG. 7 has been pressed (step S9). When the send button 535 has not been pressed (step S9/NO), the terminal control unit 130 waits until the send button 535 is pressed. When the send button 535 is pressed (step S9/YES), the terminal device 100 transmits the selected heating information to the server device 300 (step S10).

Next, the operation of the terminal control unit 130 when the determination in step S3 shown in FIG. 16 is a negative determination will be described with reference to the flowchart shown in FIG.
The terminal control unit 130 first determines whether the second process, which is the process of managing the inventory of scanned foods or ingredients, has been selected (step S11). The terminal control unit 130 determines whether the second process display 513 has been selected, and then determines whether the second process has been selected.

When the second process is selected (step S11/YES), the terminal control unit 130 displays the scan screen 540 shown in FIG. 8. For example, the terminal control unit 130 displays the scan screen 540 with the code recognition display set as the default (step S12).

Next, the terminal control unit 130 determines whether an operation to change the display of the scan screen 540 has been received by operating the switching display 541 (step S13). If an operation to change the display of the scan screen 540 has not been received (step S13/NO), the terminal control unit 130 determines whether a one-dimensional code or a two-dimensional code has been detected from the image captured by the camera 113 (step S14). Step S14 corresponds to the detection step.

If a one-dimensional code or two-dimensional code is not detected (step S14/NO), the terminal control unit 130 repeats the determination in step S14 until a one-dimensional code or two-dimensional code is detected.

When the one-dimensional code or two-dimensional code is detected (step S14/YES), the terminal control unit 130 acquires the identification data contained in the detected one-dimensional code or two-dimensional code. When the second process display 513 is selected in the judgment of step S11, the terminal control unit 130 judges that the use of the code information is inventory management of the scanned food or ingredients. The terminal control unit 130 transmits a request to acquire food management information including the food identification data and the user ID 145 to the server device 300, and acquires the food management information of the food corresponding to the identification data from the server device 300 (step S15). When the terminal control unit 130 acquires the food management information from the server device 300, it displays the acquired food management information on the confirmation screen 550 shown in FIG. 9 (step S16). This confirmation screen 550 displays information such as the expiration date and storage method contained in the food management information acquired from the server device 300. Steps S14 and S15 correspond to the first judgment procedure and the transmission procedure. Also, step S16 corresponds to the display procedure.

Next, the terminal control unit 130 determines whether the registration button 555 displayed on the confirmation screen 550 has been pressed (step S17). If the registration button 555 has been pressed (step S17/YES), the terminal control unit 130 stores the food management information as inventory management data stored in the terminal storage unit 140 (step S18). The terminal control unit 130 then displays the home screen 50 on the touch panel 117 and returns to the determination of step S1.

In addition, in step S13, if the terminal control unit 130 receives an operation to change the display of the scan screen 540 by operating the switching display 541 (step S13/YES), it changes the display mode of the scan screen 520 to the image recognition display shown in FIG. 11 and changes the operating mode to the image recognition mode.

Next, the terminal control unit 130 determines whether the scan button 549 has been pressed (step S19). If the scan button 549 has not been pressed (step S20/NO), the terminal control unit 130 waits to start processing until the scan button 549 is pressed. If the scan button 549 has been pressed (step S19/YES), the terminal control unit 130 extracts feature points from the image captured by the camera 113 and compares the extracted feature points with pre-stored feature points of the ingredients to determine the ingredients (step S20). Steps S19 and S20 correspond to the second determination procedure.

Next, the terminal control unit 130 displays the candidate ingredients determined by image recognition on the touch panel 117 (step S21). The terminal control unit 130 displays the selection screen 560 shown in FIG. 12. This selection screen 560 displays an ingredient selection field 561 and a selection field 563. After displaying the selection screen 560, the terminal control unit 130 determines whether an ingredient has been selected (step S22). If an ingredient has been selected (step S22/YES), the terminal control unit 130 sends a request to the server device 300 to obtain ingredient management information for the corresponding ingredient, and obtains the ingredient management information from the server device 300 (step S25).

Next, the terminal control unit 130 displays the acquired food ingredient management information on the confirmation screen 570 shown in FIG. 13 (step S26). This confirmation screen 570 also displays information such as the expiration date and storage method included in the food ingredient management information acquired from the server device 300.

Next, the terminal control unit 130 determines whether the registration button 573 displayed on the confirmation screen 570 has been pressed (step S27). If the registration button 573 has been pressed (step S27/YES), the terminal control unit 130 stores the food ingredient management information as inventory management data stored in the terminal storage unit 140 (step S28). The terminal control unit 130 then displays the home screen 50 on the touch panel 117 and returns to the determination of step S1.

If the determination in step S22 is negative (step S22/NO), the terminal control unit 130 determines whether or not an operation to select rescan has been received without selecting an ingredient (step S23). If rescan has been selected (step S23/YES), the terminal control unit 130 returns to the process in step S12 and displays the scan screen 540.

If the terminal control unit 130 does not accept an operation to select rescan (step S23/NO), it accepts input from the user (step S24). For example, the terminal control unit 130 displays major categories such as vegetables, meat, fish, and fruit, and minor categories such as tomatoes, lettuce, and Japanese mustard spinach on the touch panel 117, and the user inputs information into the relevant items for the major and minor categories. The terminal control unit 130 then proceeds to the process of step S23 and obtains food management information from the server device 300.

Next, the operation of the terminal control unit 130 when the determination in step S29 shown in FIG. 17 is a negative determination will be described with reference to the flowchart shown in FIG.
The terminal control unit 130 first determines whether the third process, which is the process of searching for recipes including the scanned ingredients, has been selected (step S29). The terminal control unit 130 determines whether the third process display 515 has been selected, and then determines whether the third process has been selected.

If the third process is not selected (step S29/NO), the terminal control unit 130 returns to the judgment of step S3 shown in FIG. 16. If the third process is selected (step S29/YES), the terminal control unit 130 transitions the operation mode to the image confirmation mode and displays the scan screen 540 for image recognition display shown in FIG. 14.

The operations from steps S30 to S36 shown in FIG. 18 are the same as the operations from steps S19 to S24 shown in FIG. 17, so a description of the operations from steps S30 to S36 will be omitted.
Next, when the terminal control unit 130 identifies the ingredients, it transmits a request to the server device 300 to obtain a recipe including the identified ingredients, and obtains recipe data from the server device 300 (step S37).

Next, the terminal control unit 130 displays the search result screen 590 shown in FIG. 15 based on the acquired recipe data. The acquired recipe data is displayed in a list on the search result screen 590 (step S38). Next, the terminal control unit 130 determines whether or not an operation to select a recipe has been accepted (step S39). If an operation to select a recipe has been accepted (step S39/YES), the terminal control unit 130 displays the selected recipe on the touch panel 117 (step S40). If an operation to select a recipe has not been accepted (step S39/NO), the terminal control unit 130 determines whether or not an end operation has been accepted (step S41). For example, the terminal control unit 130 determines whether or not a close button displayed on the display screen has been pressed, and determines whether or not an end operation has been accepted.

If the terminal control unit 130 receives an end operation (step S41/YES), it displays the home screen 50 on the touch panel 117 (step S42) and returns to the determination of step S1. If the terminal control unit 130 does not receive an end operation (step S41/NO), it returns to the determination of step S39.

FIG. 19 is a flowchart showing the operation of the server device 300.
The operation of the server device 300 will be described with reference to the flowchart shown in FIG.
First, the server control unit 310 judges whether or not data has been received from the terminal device 100 (step T1). If no data has been received from the terminal device 100 (step T1/NO), the server control unit 310 waits until data is received.

When the server control unit 310 receives data from the terminal device 100 (step T1/YES), it determines whether the received data includes a request to obtain heating information (step T2). When the received data includes a request to obtain heating information (step T2/YES), the server control unit 310 reads out the heating information of the corresponding food from the database 323 (step T3). The server control unit 310 transmits the read out heating information to the terminal device 100 that received the data (step T10).

If the received data does not include a request to obtain heating information (step T2/NO), the server control unit 310 determines whether the received data includes a request to obtain food management information (step T4). If the received data includes a request to obtain food management information (step T4/YES), the server control unit 310 reads the food management information of the corresponding food from the database 323 (step T5). The server control unit 310 transmits the read food management information to the terminal device 100 that received the data (step T10).

If the received data does not include a request to acquire food management information (step T4/NO), the server control unit 310 determines whether the received data includes a request to acquire ingredient management information (step T6). If the received data includes a request to acquire ingredient management information (step T6/YES), the server control unit 310 reads out the ingredient management information for the corresponding ingredient from the database 323 (step T7). The server control unit 310 transmits the read ingredient management information to the terminal device 100 that received the data (step T10).

If the received data does not include a request to acquire ingredient management information (step T6/NO), the server control unit 310 determines whether the received data includes a request to acquire recipe data (step T8). If the received data includes a request to acquire recipe data (step T8/YES), the server control unit 310 reads recipe data including the specified ingredients included in the data from the database 323 (step T9). The server control unit 310 transmits the read recipe data to the terminal device 100 that received the data (step T10).

If the received data does not include a request to obtain recipe data (step T8/NO), the server control unit 310 executes processing corresponding to the received data (step T11) and ends the processing flow.

[1-3. Effects, etc.]

As described above, the service application 143 causes the terminal control unit 130 to execute a detection procedure, a first determination procedure, a transmission procedure, and a display procedure.
The detection procedure is a procedure for detecting code information contained in an image captured by the camera 113 .
The first determination step is a step of determining the purpose of the code information based on a touch operation received on the touch panel 117 .
The transmission step is a step of transmitting, to the server device 300, a request to acquire information corresponding to the determined purpose of the code information.
The display procedure is a procedure for displaying the information received from the server device 300 .

This allows the use of the code information to be determined based on the received operation, and information corresponding to the determined use can be obtained from the server device 300 and displayed. Therefore, even when the same code information is read, information corresponding to the operation can be displayed on the display unit, improving user convenience.

In addition, the transmission procedure transmits a request to the server device 300 to obtain heating information related to cooking if the purpose of the code information determined by the first determination procedure is cooking to heat food.

As a result, if the purpose of the code information is cooking to heat food, heating information related to cooking can be obtained and displayed on the terminal device 100.

In addition, if the purpose of the code information determined by the first determination procedure is food inventory management, the transmission procedure transmits to the server device 300 a request to obtain information indicating the expiration date of the food indicated by the code information.

As a result, if the use of the code information is for food inventory management, information indicating the expiration date of the food can be obtained and displayed on the terminal device 100.

In addition, the service app 143 causes the device control unit 130 to execute a second determination procedure for determining the ingredient captured in the captured image.
The transmission step includes transmitting, to the server device 300, a request for obtaining information indicating the expiration date of the ingredient determined in the second determination step.

This allows the ingredients captured in the captured image to be determined, and information indicating the expiration date of the determined ingredients to be displayed on the terminal device 100.

(Embodiment 2)
A second embodiment will now be described.
In the above-described first embodiment, when the scan icon 53 is selected, the process selection screen 510 is displayed on the touch panel 117, and the user is allowed to select one of the first process, the second process, and the third process. In the second embodiment, when the scan icon 53 displayed on the home screen 50 shown in Fig. 4 is selected, the process selection screen 510 for selecting the first process, the second process, and the third process is not displayed, and the scan screens 520, 540 displaying code recognition or the scan screens 540, 580 displaying image recognition are displayed, and the code recognition or image recognition is performed.

When the terminal control unit 130 acquires code information using the scan screens 520 and 540 displaying code recognition, it displays a processing selection screen showing a first processing display 511 and a second processing display 513, and allows the user to select which processing to execute.
In addition, when the terminal control unit 130 determines ingredients using the scan screens 540 and 580 of the image recognition display, it displays a processing selection screen showing the second processing display 513 and the third processing display 515, and allows the user to select which processing to execute.

Even with this procedure, the same effect as in embodiment 1 described above can be obtained.

The above-described embodiment merely shows one aspect of the present invention, and any modification and application is possible within the scope of the present invention.

For example, the functions of the terminal control unit 130 may be realized by multiple processors or semiconductor chips.

The image recognition unit 153 may also identify ingredients through image recognition using AI. For example, a machine learning process is performed to learn feature quantities such as the color and shape of ingredients from a teacher's image of the ingredients, and a trained model is constructed. This trained model is stored in the device storage unit 140. When an image captured by the camera 113 is input, the image recognition unit 153 refers to the trained model to determine the ingredients contained in the captured image. The trained model may also be stored in the server storage unit 320, and the server device 300 may determine the ingredients. In this case, when an image captured by the camera 113 is input, the image recognition unit 153 transmits the input image to the server device 300 and obtains the result of determining the ingredients from the server device 300.

The components shown in FIG. 2 are merely examples, and the specific implementation form is not particularly limited. In other words, it is not necessary to implement hardware that corresponds to each component individually, and it is of course possible to implement a configuration in which a single processor executes a program to realize the functions of each component. Also, some of the functions realized by software in the above-mentioned embodiment may be hardware, or some of the functions realized by hardware may be software.

For example, the units of operation shown in the flowcharts of Figures 16-18 are divided according to the main processing content in order to make it easier to understand the operation of the terminal device 100, and the present invention is not limited by the manner in which the processing units are divided or their names. Depending on the processing content, the processing units may be further divided into more step units. Also, one step unit may be divided so as to include even more processing. The order of the steps may be changed as appropriate as long as it does not interfere with the spirit of the present invention. The same applies to the flowchart showing the operation of the server device 300 shown in Figure 19.

The above-described embodiments are intended to illustrate the technology disclosed herein, and various modifications, substitutions, additions, omissions, etc. may be made within the scope of the claims or their equivalents.

As described above, the program and terminal device of the present invention can be used to provide cooking information to users.

1 Service providing system 10 Network 100 Terminal device 111 Terminal communication I/F
REFERENCE SIGNS LIST 113 Camera 115 Display unit 117 Touch panel 130 Terminal control unit 140 Terminal storage unit 141 Control program 143 Service application 150 Terminal processor 151 Code detection unit 153 Image recognition unit 155 Data acquisition unit 200 Cooking appliance 201 Cooking appliance communication I/F
203 Operation unit 210 Cooking control unit 220 Heat cooking unit 300 Server device 301 Server communication I/F
310 Server control unit 320 Server storage unit 323 Database 323a User database

Claims (9)

On the computer,
a detection step of detecting code information included in an image captured by the image capturing unit;
a first determination step of determining a purpose of the code information based on an operation received by an operation unit;
a transmission step of transmitting a request for acquiring information corresponding to the determined purpose of the code information to a server device;
a display step of displaying the information received from the server device;
Run the command,
The sending step is a program for sending a request to the server device to obtain heating information related to cooking when the purpose of the code information determined in the first determination step is cooking for heating food. On the computer,
a detection step of detecting code information included in an image captured by the image capturing unit;
a first determination step of determining a purpose of the code information based on an operation received by an operation unit;
a transmission step of transmitting a request for acquiring information corresponding to the determined purpose of the code information to a server device;
a display step of displaying the information received from the server device;
Run the command,
The sending step is a program for sending to the server device a request to obtain information indicating the expiration date of the food indicated by the code information when the use of the code information determined by the first determination step is food inventory management. On the computer,
a detection step of detecting code information included in an image captured by the image capturing unit;
a first determination step of determining a purpose of the code information based on an operation received by an operation unit;
A second determination step of determining the food ingredient captured in the captured image;
a transmission step of transmitting a request for acquiring information corresponding to the determined purpose of the code information to a server device;
a display step of displaying the information received from the server device;
Run the command,
The sending step includes sending to the server device a request for information indicating the expiration date of the ingredient determined in the second determination step. A display unit;
The Communications Department and
The photography department and
a code detection unit that detects code information included in the image captured by the image capture unit;
An operation unit that accepts operations;
a control unit that determines a use of the code information based on the operation received by the operation unit, and transmits a request for acquiring information corresponding to the determined use to a server device via the communication unit;
The control unit causes the display unit to display the information received from the server device ,
When the determined purpose of the code information is cooking for heating food, the terminal device transmits a request to the server device to obtain heating information related to cooking . A display unit;
The Communications Department and
The photography department and
a code detection unit that detects code information included in the image captured by the image capture unit;
An operation unit that accepts operations;
a control unit that determines a use of the code information based on the operation received by the operation unit, and transmits a request for acquiring information corresponding to the determined use to a server device via the communication unit;
The control unit causes the display unit to display the information received from the server device,
When it is determined that the purpose of the code information is food inventory management, the terminal device transmits a request to the server device to obtain information indicating the expiration date of the food indicated by the code information . A display unit;
The Communications Department and
The photography department and
a code detection unit that detects code information included in an image captured by the image capture unit;
An operation unit that accepts operations;
a control unit that determines a use of the code information based on the operation received by the operation unit, and transmits a request for acquiring information corresponding to the determined use to a server device via the communication unit;
The control unit causes the display unit to display the information received from the server device,
A terminal device determines the food material captured in the captured image, and transmits a request to the server device for information indicating the expiration date of the determined food material . a terminal device including a display unit, a communication unit, an image capture unit, a code detection unit that detects code information included in an image captured by the image capture unit, an operation unit that receives an operation, and a control unit that determines a use of the code information based on the operation received by the operation unit and transmits a request to obtain information corresponding to the determined use to a server device via the communication unit;
the server device transmits information corresponding to the acquisition request to the terminal device;
The control unit causes the display unit to display the information received from the server device ,
When the determined purpose of the code information is cooking for heating food, the service providing system transmits a request to the server device to obtain heating information related to cooking . a terminal device including a display unit, a communication unit, an image capture unit, a code detection unit that detects code information included in an image captured by the image capture unit, an operation unit that receives an operation, and a control unit that determines a use of the code information based on the operation received by the operation unit and transmits a request to obtain information corresponding to the determined use to a server device via the communication unit;
the server device transmits information corresponding to the acquisition request to the terminal device;
The control unit causes the display unit to display the information received from the server device,
When the determined purpose of the code information is food inventory management, the service providing system transmits a request to the server device to obtain information indicating the expiration date of the food indicated by the code information. a terminal device including a display unit, a communication unit, an image capture unit, a code detection unit that detects code information included in an image captured by the image capture unit, an operation unit that receives an operation, and a control unit that determines a use of the code information based on the operation received by the operation unit and transmits a request to obtain information corresponding to the determined use to a server device via the communication unit;
the server device transmits information corresponding to the acquisition request to the terminal device;
The control unit causes the display unit to display the information received from the server device,
The service providing system determines the food material captured in the captured image, and transmits a request to the server device to obtain information indicating the expiration date of the determined food material.

Images