JP7565246B2 - Heating state identification device, heating control device, heating control method, heating state identification system, and heating control system - Google Patents

Description

Embodiments of the present invention relate to a heating state identification device, a heating control device, a heating control method, a heating state identification system, and a heating control system.

Traditionally, restaurant staff have judged the amount of heat to use and strived to provide customers with food that is cooked to the highest possible quality. For example, there is a cooking method in which the temperature of the pot is adjusted to prevent the food from boiling. However, the temperature adjustment of the heating device that heats the pot etc. can depend on the experience of the staff.

JP 2017-133722 A

The problem that the present invention aims to solve is to provide a heating state identification device, a heating control device, a heating control method, a heating state identification system, and a heating control system that enable cooking with more appropriate heating control regardless of the cook's experience.

The heating state identification device according to one embodiment includes: an acquisition unit that acquires a plurality of time-series images of the inside of a container having a heated object and a liquid in a time series;
an identification unit that identifies a plurality of in-container states of the liquid using time-series feature amounts acquired based on a plurality of time-series images;
Equipped with.

The image processing apparatus further includes a feature generating unit having a convolutional neural network that generates image feature amounts for each of the plurality of time-series images, arranges the image feature amounts in a time series, and generates the time-series image feature amounts;
The identification unit is a recurrent neural network that identifies multiple heating states based on the time-series image features generated by the convolutional neural network.
It may have.

A learning unit that performs machine learning using a learning set including a correct answer label corresponding to the state of the heated object and the liquid and a time-series image feature generated by the convolutional neural network,
The identification unit may identify the state based on the results of machine learning.

The heating control device may be a heat power control unit that controls the heat power of the heating device that heats the container based on the identification result of the identification unit of the above-mentioned heating state identification device.

The heat control unit may also perform control to change the time for which a specified state is maintained depending on the specified state before the liquid boils.

When the identification unit identifies a state in which an object to be heated has been placed in the container, regardless of the liquid state of the liquid, the heat control unit may cause the display device to display an image encouraging the user to discard the object to be heated in the container after a predetermined time has elapsed since the time when the identification unit identified a state in which an object to be heated has been placed in the container.

The heating control method according to one embodiment includes: acquiring a plurality of time-series images of an interior of a container having an object to be heated and a liquid in a time-series manner;
an identification step of acquiring time-series image features based on a plurality of time-series images, and identifying a plurality of heating states in the liquid using the time-series image features;
Equipped with.

a heating power control step of controlling the heating power of a heating device for heating the container based on the identification result of the identification step of the heating state identification method;
The present invention may also include:

According to one embodiment, the heating state identification system includes: an imaging device that images the inside of a container having a heated object and a liquid from an upper portion of the container;
an acquisition unit that acquires a plurality of images captured in time series by an imaging device;
an identification unit that identifies a plurality of heating states in the liquid using time-series image feature amounts acquired based on a plurality of time-series images;
Equipped with.

The heating control system includes a heat power control unit that controls the heat power of a heating device that heats a container based on the identification result of the identification unit of the heating state identification system described above;
The present invention may also include:

The present invention provides a heating control device, a heating control method, and a heating control system that enable cooking with more appropriate heating control regardless of the cook's experience.

FIG. 1 is a block diagram showing a configuration of a heating control system according to an embodiment of the present invention. FIG. 13 is a diagram showing an example of the position of an image capturing device capturing an image of a cooking container. FIG. 4 is a diagram showing an example of the configuration of a feature generating unit. A table that explains in words the state of the heated object and liquid. 13 is a flowchart showing an example of a learning process of a learning unit. 5 is a flowchart showing an example of control by a thermal power control unit. 11 is a table showing examples of correct labels corresponding to human operations. FIG. 13 is a diagram showing an example in which a state in which the fourth state is followed by the third state occurs multiple times. FIG. 13 is a block diagram showing the configuration of a cooking instruction device according to a second embodiment. 10 is a flowchart showing an example of control by a thermal power control unit according to the second embodiment.

Hereinafter, a heating state identification device, a heating control device, a heating control method, a heating state identification system, and a heating control system according to embodiments of the present invention will be described in detail with reference to the drawings. Note that the embodiment described below is one example of an embodiment of the present invention, and the present invention is not to be interpreted as being limited to these embodiments. Furthermore, in the drawings referred to in this embodiment, identical parts or parts having similar functions are given the same or similar symbols, and repeated explanations thereof may be omitted. Furthermore, the dimensional ratios of the drawings may differ from the actual ratios for the convenience of explanation, and some components may be omitted from the drawings.

First Embodiment
Fig. 1 is a block diagram showing the configuration of a heating control system 1 according to a first embodiment. The heating control system 1 in Fig. 1 is a system capable of controlling the heat of a heating device. As shown in Fig. 1, the heating control system 1 according to the present embodiment is configured to include a heating device 10, a cooking container 20, an image capture device 30, a display device 40, an operation unit 50, and a heating control device 60.

Figure 2 is a diagram showing an example of the position of the imaging device 30 photographing from above the inside of the cooking vessel 20 placed on the heating device 10. As shown in Figure 2, the heating device 10 is, for example, a gas stove, and has a burner as a heat generating part on the upper surface of the housing. In addition, the flame power of the burner (adjustment of the amount of heat generated) can be adjusted by rotating the burner operation button using a motor. Note that the heating device 10 according to this embodiment is a gas stove, but is not limited to this. For example, it may be an electric stove.

Cooking container 20 is, for example, a pot, and stores therein foodstuffs to be heated and cooking water, which is a mixture of water and seasonings.
The image capturing device 30 is, for example, a camera, and is capable of capturing images showing the state inside the cooking container 20 in chronological order from above the cooking container 20. The image capturing device 30 outputs, for example, three types of digital image data, a red image (R image), a green image (G image), and a blue image (B image), to the heating control device 60. Each of the three types of digital image data is composed of, for example, 1080 vertical × 1920 horizontal pixels.

As shown again in FIG. 1, the display device 40 is, for example, a monitor. The display device 40 displays the image generated by the heating control device 60.

The operation unit 50 is composed of, for example, a keyboard, a pointing device, etc. It can also be integrated into the display device 40 as a touch panel. The operation unit 50 outputs instruction signals in response to operations by the user who uses the heating control device 60.

The heating control device 60 is a device capable of controlling the heating power of the heating device 10, and includes an acquisition unit 600, a storage unit 602, a feature generating unit 603, a learning unit 604, an identification unit 607, a heating power control unit 608, and an image generating unit 610. Such a heating control device 60 is realized, for example, by a personal computer. That is, the heating control device 60 is configured to include, for example, a CPU (Central Processing Unit). In addition, the heating state identification device 61 according to this embodiment includes the acquisition unit 600, the feature generating unit 603, the learning unit 604, and the identification unit 607, and the heating state identification system 62 according to this embodiment includes the imaging device 30 and the heating state identification device 61. Note that the heating state identification device 61 according to this embodiment includes the learning unit 604, but is not limited thereto. For example, the heating state identification device 61 may be configured without including the learning unit 604.

The acquisition unit 600 can also acquire time-series images captured by the image capture device 30 and store them in the storage unit 602. The acquisition unit 600 is also connected to a network, for example, and can acquire images from a server or the like via the network.

The storage unit 602 is configured, for example, with a HDD (hard disk drive) or SSD (solid state drive). The storage unit 602 can also store images acquired by the acquisition unit 600 for learning purposes. The storage unit 602 also stores various programs for executing heating control. As a result, the heating control device 60 configures each processing unit by, for example, executing a program stored in the storage unit 602. The storage unit 602 also stores a machine learning model for identification. Note that in this embodiment, a machine learning model that has been trained may be referred to as a trained model.

The feature generating unit 603 generates image features in a time series from each of the time series images acquired by the acquiring unit 600, for example. The learning data is data in which the time series image features generated by the feature generating unit 603 are associated with correct answer labels corresponding to the state of the heated object and liquid in the cooking container 20. The feature generating unit 603 will be described in detail later.

The learning unit 604 performs machine learning, for example supervised learning, using the learning data stored in the memory unit 602. The learning unit 604 performs machine learning, for example, using a neural network. The learning unit 604 may also perform calculations outside the heating control device 60, for example, in a computer with higher calculation power for learning. The learned model (trained model) is stored in the memory unit 602, for example, via a network, and used by the identification unit 607. Note that, in this embodiment, the learning unit 604 performs calculations in a computer outside the heating control device 60, but is not limited to this. For example, the learning unit 604 may be provided in the heating control device 60. In this case, it is also possible to immediately reflect data obtained on-site in the learning.

The machine learning model that the learning unit 604 learns by machine learning is not limited to a neural network. For example, a discriminant function or the like may be learned as a machine learning model by a general clustering method such as the K-means method. Details of the learning unit 604 will be described later.

The identification unit 607 uses the results of machine learning in the learning unit 604 to identify the predetermined state of the liquid in the cooking vessel 20. The identification unit 607 according to this embodiment uses a neural network trained in the learning unit 604. This neural network is capable of identifying multiple heating states, including a predetermined state before the liquid in the cooking vessel 20 boils, using time-series features generated by the feature generation unit 603 based on multiple time-series images captured by the imaging device 30. Details of the identification unit 607 will be described later.

The heat power control unit 608 controls the heat power of the heating device 10 based on the identification result of the identification unit 607. For example, the heat power control unit 608 controls the rotation of the motor of the heating device 10 to control the heat power of the heating device 10. Details of the heat power control unit 608 will also be described later.

The image generating unit 610 generates an image showing the state inside the cooking container 20 based on the identification result of the identifying unit 607, and outputs the image to the display device 40. The image generating unit 610 may also generate an image showing an operation instruction based on the identification result of the identifying unit 607.

Here, the feature generating unit 603 and the learning unit 604 will be described in detail with reference to FIG.
FIG. 3 is a diagram showing an example of the configuration of the feature generating unit 603. As shown in FIG. 3, the feature generating unit 603 has a plurality of image feature extracting units 603a. The image feature extracting unit 603a is, for example, a convolutional neural network (CNN, hereinafter sometimes simply referred to as CNN). Note that, in this embodiment, a plurality of image feature extracting units 603a are used, but this is not limited thereto. For example, images t1 to tn may be input in sequence to one image feature extracting unit 603a, and image feature amounts t1 to tn may be generated in sequence.

In this CNN, neurons in each layer are grouped into filters, and each filter detects different overlapping features in the input data. The multi-layer neural network architecture in the convolutional neural network has, for example, multiple convolutional layers, multiple sampling layers, and multiple fully connected layers. In addition, this CNN can be trained using a pre-trained CNN (e.g., ImageNet pre-trained net) that is trained using existing labeled images.
That is, the feature extraction unit 603a fixes the convolutional neural network, feeds forward the input image, and generates the value output from the intermediate layer as the feature.

Each of the multiple feature extraction units 603a receives images t1 to tn captured in time series. For example, one image uses three digital images, a red image (R image), a green image (G image), and a blue image (B image), each reduced to 224 x 224 pixels. In this embodiment, the three primary colors R, G, and B are each stored as an 8-bit (256 gradations) value. As a result, 224 x 224 x 3 pixel data is input to the first convolution layer of the CNN. Then, time-series image features t1 to tn are generated. For example, n is 60, and 60 time-series images captured per second are input to the learning unit 604 or the identification unit 607. In addition, the image features generated by the feature extraction unit 603a are, for example, 1024-dimensional. In this way, the CNN generates image features t1 to tn for each of the images t1 to tn. The feature generation unit 603 then outputs the image features t1 to tn arranged in chronological order.

The learning data will be described in detail with reference to FIG.
FIG. 4 is a table that describes the state of the object to be heated and the liquid in text. The state of the object to be heated and the liquid in the cooking container 20 changes in a time series as the heating time passes. That is, as the heating of the cooking container 20 progresses, the state changes in a time series, for example, from the first state to the fourth state. For example, the first state is "a state where the liquid is not boiling and no operation is being performed". The second state is "a state where the liquid level is stable and bubbles are rising here and there". The third state is "a state where the liquid level is raised, but bubbles are not generated or are not noticeable". The fourth state is "a state where the liquid is boiling". For example, the first state to the third state show the state before boiling. In the learning data, a label indicating the first state is assigned as a correct answer label to the time-series image obtained by the acquisition unit 600 or the time-series image feature obtained by the feature amount generation unit 603 in the first state, a label indicating the second state is assigned as a correct answer label to the time-series image obtained by the acquisition unit 600 or the time-series image feature obtained by the feature amount generation unit 603 in the second state, a label indicating the third state is assigned as a correct answer label to the time-series image obtained by the acquisition unit 600 or the time-series image feature obtained by the feature amount generation unit 603 in the third state, and a label indicating the fourth state is assigned as a correct answer label to the time-series image obtained by the acquisition unit 600 or the time-series image feature obtained by the feature amount generation unit 603 in the fourth state. These labels can be assigned manually, for example. These labeled time-series images or time-series image feature amounts are stored in the storage unit 602 as learning data.

Returning to FIG. 3, the learning unit 604 will be described. This learning unit 604 uses the above-mentioned learning data to learn a neural network, which is an example of a machine learning model. The neural network trained by the learning unit 604 learns, for example, labeled time-series image features and learns to output labels (first to fourth states) for the input time-series image features. For example, this neural network is a neural network (LSTM: Long Short-term Memory) in which layers having multiple nodes are connected in multiple stages. LSTM is a form of a recurrent neural network that learns time-series data.

For example, in this neural network, the connection coefficients between each node are initialized, and labeled time-series image features are input in order. The neural network is then generated by processing such as backpropagation, which corrects the connection coefficients so as to reduce the error between the output of the neural network and the labels assigned to the input time-series image features. For example, a trained neural network is generated by performing processing such as backpropagation to minimize a predetermined loss function. By repeating the above-mentioned processing, the neural network is able to better reproduce and output the labels for the input features. Note that the method of training the neural network is not limited to the above-mentioned method, and any known technology can be applied.

The identification unit 607 has a neural network that has undergone the above-mentioned training. This neural network is now able to distinguish between multiple pre-boiling states, which was previously difficult to distinguish. For example, in the first state, "not boiling and no operation being performed," there is no variation in the time-series image feature amount, and the neural network can identify this as the first state.

For example, in the second state, "where the liquid level is stable and bubbles are rising in places," the time-series fluctuations in the values of the pixel data corresponding to the bubbles are captured as time-series image features. In this way, for example, when time-series fluctuations in the image features corresponding to bubbles are captured, the neural network can identify the second state.

For example, in the third state, "the liquid surface is raised, but no bubbles are generated or are not noticeable," the time series fluctuations in the pixel data values corresponding to the raised liquid surface are captured as time series image features. In this way, for example, when the time series fluctuations in the image features corresponding to the raised liquid surface are captured, the neural network can identify the third state.

Furthermore, for example, in the fourth state, "boiling state," the time series fluctuations in the values of pixel data corresponding to the rising of the liquid surface and bubbles can be captured from the entire area of the image. In this case, the time series fluctuations in the values of pixel data corresponding to bubbles are greater than in the third state. In this way, the time series fluctuations in the values of pixel data corresponding to the rising of the liquid surface are captured as time series image features. As a result, for example, when larger time series fluctuations in the values of pixel data corresponding to the liquid surface and bubbles are captured as time series image features, the neural network can identify the fourth state.

In this way, minute changes before boiling can be captured by time-series image features acquired based on multiple time-series images taken in time series of the inside of the cooking vessel 20 containing the heated object and liquid. This makes it possible to identify multiple heating states of the liquid using these time-series image features. Note that in this embodiment, the heating states are classified into four categories, but this is not limiting. For example, five or ten categories may be used.

Here, an example of the learning process of the learning unit 604 will be described with reference to FIG. 5. FIG. 5 is a flowchart showing an example of the learning process of the learning unit 604. As shown in FIG. 5, first, the learning unit 604 acquires a set of learning data from the storage unit 602 (step S100). If the learning data is a time-series image, the feature generating unit 603 is used to convert it into a time-series image feature. Next, the learning unit 604 starts learning the neural network (step S102). In this case, learning is performed by providing a time-series image feature as an input and providing any one of labels 1 to 4 corresponding to the time-series image feature as a teacher signal (step S104). That is, learning is performed by backpropagation, which corrects the connection coefficients so as to reduce the error between the label corresponding to the input feature and the output value of the neural network.

Next, the learning unit 604 determines whether or not a learning end condition is satisfied (step S106).
If the termination condition is not satisfied (NO in step S106), the process is repeated from step S104. The termination condition is, for example, the number of times learning has been performed.

On the other hand, if the termination condition is met (YES in step S106), the entire process is terminated. In this way, a neural network is trained to output a state for time-series image features obtained from multiple time-series images.

Here, an example of the processing program of the heat power control unit 608 will be described. The processing program of the heat power control unit 608 is programmed with, for example, the duration of the first to fourth states for each type of food. Here, when cooking without boiling, the heat power control unit 608, for example, sets the time of the third state just before boiling to the longest, and controls the heating and cooling states of the heated object and liquid.

For example, the heat control unit 608 uses the identification result of the identification unit 607 to control the strength of the heat of the heating device 10 and the time for which the heat is maintained so as to transition between programmed states.

An example of control by the fire power control unit 608 will be described with reference to FIG. 6. FIG. 6 is a flowchart showing an example of control by the fire power control unit 608. As shown in FIG. 6, first, the acquisition unit 600 acquires a plurality of images in time series from the image capture device 30 (step S200).

Next, the feature generating unit 605 generates time-series image feature values from the 60×60×3 (60 frames) of time-series pixel data, inputs the generated feature values to the classifier 607, and classifies the states of a plurality of time-series images (step S202).
Next, the heating power control unit 608 controls the heating power of the heating device 10 according to the state identified by the identification unit (step S204). For example, when the current state is to be maintained, the heating power of the heating device 10 is controlled so as not to change the heating power, and when the state is to be heated to the next state, the heating power of the heating device 10 is increased. On the other hand, when the state is to be cooled to the next state, the heating power of the heating device 10 is decreased.

Next, the heat power control unit 608 determines whether the programmed cooking has finished (step S206), and if not finished (No in step S206), repeats the process from step S200. On the other hand, if finished (Yes in step S206), the heating device 10 is stopped and the entire process ends. When controlling heating using a thermometer, uneven temperature occurs depending on the position of the heated object and the state of convection. In contrast, identification using time-series images uses information on the state changes of the entire image, so the state can be identified with higher accuracy. This allows heating control by the heat power control unit 608 to be performed with higher accuracy.

As described above, according to this embodiment, the learning unit 604 performs machine learning to determine a predetermined state before the liquid boils using multiple time-series images of a container containing a heated object and liquid taken in time series. This enables the identification unit 607 to capture minute changes in the surface image of the liquid as time-series fluctuations in image features based on the learning results of the learning unit 604, and to identify the predetermined state before boiling. In addition, since the identification unit 607 identifies the predetermined state before boiling, the heat control unit 608 can control the heating to cook the container containing the heated object and liquid without boiling, for example.

(First Modification of the First Embodiment)
The heating control system 1 according to the first embodiment identifies the heating state of the cooking container 20 by the heating device 10, but the heating control system 1 according to the first modification of the first embodiment differs from the heating control system 1 according to the first embodiment in that it can also identify human operation on the cooking container 20. The differences from the first embodiment will be described below.

Figure 7 is a table showing examples of correct labels corresponding to the human operations of the cook. For example, the fifth state is "a state in which the cook is stirring with a utensil regardless of the liquid state." The sixth state is "a state in which the cook is skimming off the scum using a utensil regardless of the liquid state." The seventh state is "a state in which ingredients are being added regardless of the liquid state."

That is, in the heating control system 1 according to the modified example of the first embodiment, a machine learning model to be used in the identification unit 607 is trained using time-series images of the fifth to seventh states in addition to the first to fourth states. In this case, for example, a correct answer label of 5 is assigned to the fifth state, 6 to the sixth state, and 7 to the seventh state, and the machine learning model to be used in the identification unit 607 is trained.

This also makes it possible to give priority to detecting the actions of the cook who is aware of the state of the pan and operating it. For example, if the identification unit 607 identifies the seventh state, after a predetermined time has elapsed since the time the seventh state was identified, the heat power control unit 608 causes the display device 40 to display an image encouraging the user to discard the heated object in the cooking container 20. This allows the quality of the heated object in the cooking container 20 to be maintained. In this way, the identification unit 607 can use multiple time-series images captured in chronological order to identify any of the fifth to seventh states, in addition to the first to fourth states.

(Second Modification of the First Embodiment)
The heating control system 1 according to the second modified example of the first embodiment further has a function of issuing a warning and stopping heating when the heating state of the cooking container 20 reaches a predetermined state.

Figure 8 shows an example in which the fourth state, "boiling," occurs multiple times, followed by the third state, "the liquid level is raised, but no bubbles are produced or noticeable."

As shown in FIG. 8, in the heating control system 1, when performing heating control that stops at the third state, there is a risk of the system moving into the fourth state. In this way, when the number of times that the fourth state occurs reaches, for example, a predetermined number of times, 3 or more, a warning is issued on the display device 40 and heating is stopped.

Some heated items may contain items whose quality may deteriorate if boiled for a long period of time. In this way, by issuing a warning based on the number of boils, it is possible to encourage the user to check the quality, and depending on the quality of the heated items, it is also possible to have the user discard the items.

(Third Modification of the First Embodiment)
The heating control device 60 according to the third modified example of the first embodiment further includes a liquid temperature measuring unit (not shown) that measures the liquid temperature in the cooking container 20, and an air pressure measuring unit (not shown) that measures the atmospheric pressure. The identification unit 607 identifies a plurality of heating states in the liquid using time-series image feature amounts acquired based on a plurality of time-series images and at least one of the liquid temperature measured by the liquid temperature measuring unit and the air pressure measured by the air pressure measuring unit. For example, the identification unit 607 can correct the identification result of a plurality of heating states in the liquid identified using the time-series image feature amounts acquired based on a plurality of time-series images, based on the result measured by at least one of the liquid temperature measuring unit and the air pressure measuring unit.

The heat power control unit 608 controls the heat power of the heating device 10 that heats the cooking container 20 using the identification result of the identification unit 607, which is based on the results measured by at least one of the liquid temperature measurement unit and the air pressure measurement unit. For example, the air pressure and boiling point vary depending on the altitude at which the heating control device 60 is installed. The identification unit 607 can correct such environmental fluctuations based on the results measured by at least one of the liquid temperature measurement unit and the air pressure measurement unit, allowing the heating control device 60 to perform control with higher precision.

(Fourth Modification of the First Embodiment)
The heating control device 60 according to the fourth modification of the first embodiment further includes a time measurement unit (not shown) that measures the time of a plurality of heating states, etc. The time measurement unit measures the elapsed time of each state (e.g., first to fourth states (FIG. 4), fifth to seventh states (FIG. 7) described later).

The heat control unit 608 controls the heat of the heating device 10 that heats the cooking container 20 based on the identification result of the identification unit 607 and the time that has elapsed since the heated object was added by the time measurement unit. For example, it can stop heating after simmering for a certain period of time and switch to keeping warm, etc. This allows the heat control unit 608 to perform control such as stopping heating at a specific time after the food ingredient (e.g. meat) is added.

Second Embodiment
The heating control system 1 according to the second embodiment differs from the heating control system 1 according to the first embodiment in that the neural network used by the recognition unit is changed depending on the shape of the cooking vessel 20. The differences from the first embodiment will be described below.

Figure 9 is a block diagram showing the configuration of a heating control system 1 according to the second embodiment. As shown in Figure 9, the heating control system 1 according to this embodiment is further configured to include a shape identification unit 612 of the heating control device 60.

There are multiple different shapes of cooking vessels 20, such as round and square. The learning unit 604 trains the machine learning model used by the shape identification unit 612 for shape identification. In this case, the input data is image data photographed from above the cooking vessel 20, and learning data with shape as the teacher category is used.

The machine learning model for the identification unit 607 is also trained using training data 1, which is an image of a round cooking vessel 20, and training data 2, which is an image of a square cooking vessel 20. That is, the identification unit 607 has a machine learning model 1 of a learning result 1 trained with training data 1, and a machine learning model 2 of a learning result 2 trained with training data 2.

Convection in a round cooking vessel 20 occurs with the center of the vessel at the center of the round pot. On the other hand, convection in a square cooking vessel 20 tends to shift the center of convection to a position influenced by the heating position. For this reason, the classification accuracy can be improved by selectively using machine learning model 1 and machine learning model 2 depending on the shape of the cooking vessel 20.

An example of control of the fire power control unit 608 according to the second embodiment will be described with reference to FIG. 10. FIG. 10 is a flowchart showing an example of control of the fire power control unit 608 according to the second embodiment. As shown in FIG. 12, first, the acquisition unit 600 acquires an image from the imaging device 30 (step S300).

Next, the shape identification unit 612 converts the data into 60 x 60 x 3 pixel data, inputs the data into a machine learning model used in the shape identification unit 612, identifies the state of the image, and determines whether the shape is round or not (step S302).
Next, if the shape is round (YES in step S302), the identification unit 607 uses machine learning model 1 of learning result 1, and if the shape is square (NO in step S302), the identification unit 607 uses machine learning model 2 of learning result 2.

As described above, according to this embodiment, the neural network used in the identification unit 607 is changed depending on the shape of the cooking container 20. This makes it possible to identify the state inside the container with higher accuracy even when the state inside the container differs depending on the shape of the cooking container 20.

The heating control device 60 and at least a part of the heating control system 1 described in the above embodiment may be configured as hardware or software. When configured as software, a program that realizes at least a part of the functions of the heating control device 60 and the heating control system 1 may be stored on a recording medium such as a flexible disk or CD-ROM, and may be read and executed by a computer. The recording medium is not limited to removable ones such as magnetic disks and optical disks, but may be fixed recording media such as a hard disk unit or memory.

The heating control device 60 and a program that realizes at least some of the functions of the heating control system 1 may be distributed via a communication line (including wireless communication) such as the Internet. Furthermore, the program may be encrypted, modulated, or compressed and distributed via a wired or wireless line such as the Internet, or stored on a recording medium.

Although several embodiments have been described above, these embodiments are presented only as examples and are not intended to limit the scope of the invention. The novel devices, methods, and programs described in this specification can be embodied in various other forms. Furthermore, various omissions, substitutions, and modifications can be made to the forms of the devices, methods, and programs described in this specification without departing from the spirit of the invention.

1: Cooking system, 10: Heating device, 20: Cooking container, 30: Photography device, 40: Display device, 60: Heating control device, 61: Heating state identification device, 62: Heating state identification system, 600: Acquisition unit, 603: Feature generation unit, 604: Learning unit, 606: Classifier learning unit, 607: Identification unit, 608: Heating power control unit, 612: Shape identification unit.

Claims (11)

An acquisition unit that acquires a plurality of time-series images obtained by taking time-series images of the inside of a container having an object to be heated and a liquid;
a classification unit that classifies a pre- boiling state of the liquid into a plurality of states using time-series image feature amounts acquired based on the plurality of time-series images;
a heat control unit that controls heat without boiling using the identification result of the identification unit;
A heating control device comprising: A feature generating unit that generates image features for each of the plurality of time-series images, arranges the image features in time series, and generates the time-series image features, the feature generating unit having a convolutional neural network,
The identification unit includes a recurrent neural network that identifies a plurality of heating states of the liquid, including a plurality of states before boiling , based on the time -series image feature amount generated by the convolutional neural network,
The heating control device of claim 1 . A learning unit that performs machine learning using a learning set including a correct answer label corresponding to the state of the heated object and the liquid and a time-series image feature generated by the convolutional neural network,
The heating control device according to claim 2 , wherein the identification unit identifies a state based on a result of the machine learning. The learning unit performs the machine learning using different learning sets depending on the shape of the container,
The heating control device according to claim 3 , wherein the identification unit identifies the state based on the result of the machine learning according to the shape of the container. The heating control device according to claim 1 , wherein the heat control unit controls heat of a heating device that heats the container. The heating control device according to claim 5 , wherein the heat power control unit controls the time for which the predetermined state is maintained based on the duration of each predetermined state before the liquid boils, which is predetermined for each type of food . The heating control device according to claim 5 or 6, wherein when the identification unit identifies a state in which the heated object is being added, regardless of the liquid state of the liquid, the heat power control unit causes a display device to display a display image encouraging the user to discard the heated object in the container after a predetermined time has elapsed since the time when the identification unit identified a state in which the heated object is being added. An acquisition step of acquiring a plurality of time-series images of the inside of a container having an object to be heated and a liquid in a time series manner;
a classification step of classifying pre- boiling states of the liquid into a plurality of states using time-series image feature amounts acquired based on the plurality of time-series images;
a heat control step of controlling heat without boiling the water by using the identification result of the identification step ;
A heating control method comprising: The heating control method according to claim 8 , wherein the heating power control step includes controlling a heating power of a heating device that heats the container. An imaging device that images the inside of a container from an upper portion of the container having an object to be heated and a liquid;
an acquisition unit that acquires a plurality of time-series images captured by the imaging device in a time series;
a classification unit that classifies a pre- boiling state of the liquid into a plurality of states using time- series image feature amounts acquired based on the plurality of time-series images;
a heat control unit that controls heat without boiling using the identification result of the identification unit;
A heating control system comprising: The heating control system according to claim 10 , wherein the heat power control unit controls a heat power of a heating device that heats the container.

Images