JP7706076B2 - Teacher data creation system and teacher data creation method - Google Patents

Description

This disclosure relates to a teacher data creation system and a teacher data creation method.

Technologies related to object recognition and identification have been proposed in the past. Patent Document 1 discloses an information processing device capable of efficiently recognizing different types of products for sale.

JP 2014-52810 A

The present disclosure provides a teacher data creation system and the like that can create teacher data used to build a machine learning model.

A teacher data creation system according to one aspect of the present disclosure is a teacher data creation system that creates teacher data used to build a machine learning model that identifies objects, and includes an input unit in which a user inputs label information for the object, a capture unit that captures a video in which the object appears, a display unit that displays the captured video, and a control unit that displays a guide frame within the displayed video to instruct the user to position the object within the guide frame, and the control unit creates the teacher data by linking the input label information to a still image in which the object appears that is captured by the capture unit after the instruction is given, and changes the display mode of the guide frame each time the still image is captured.

A teacher data creation system according to one embodiment of the present disclosure can create teacher data used to build a machine learning model.

FIG. 1 is a block diagram showing a functional configuration of a warehousing/shipment management system according to an embodiment. FIG. 2 is an external view of a refrigerator and an incoming/outgoing management device provided in the incoming/outgoing management system according to the embodiment. FIG. 3 is a flowchart of an operation example 1 of the incoming/outgoing management system according to the embodiment. FIG. 4 is a diagram showing an example of a moving image captured by the imaging section. FIG. 5 is a diagram showing an example of food management information. FIG. 6 is a sequence diagram of an operation example 2 of the incoming/outgoing management system according to the embodiment. FIG. 7 is a diagram showing an example of a display screen showing the contents of food management information. FIG. 8 is a sequence diagram of an operation example 3 of the incoming/outgoing management system according to the embodiment. FIG. 9 is a diagram showing an example of an input screen for label information and shape information. FIG. 10A is a diagram showing a first display example of a moving image and a guide frame. FIG. 10B is a diagram showing a second display example of a moving image and a guide frame. FIG. 10C is a diagram showing a third display example of a moving image and a guide frame. FIG. 10D is a diagram showing a fourth display example of a moving image and a guide frame. FIG. 10E is a diagram showing a fifth display example of a moving image and a guide frame. FIG. 10F is a diagram showing a sixth display example of a video and a guide frame.

The following describes the embodiments in detail with reference to the drawings. Note that the embodiments described below are all comprehensive or specific examples. The numerical values, shapes, materials, components, component placement and connection forms, steps, and order of steps shown in the following embodiments are merely examples and are not intended to limit the present disclosure. In addition, among the components in the following embodiments, components that are not described in an independent claim are described as optional components.

Note that each figure is a schematic diagram and is not necessarily a precise illustration. In addition, in each figure, the same reference numerals are used for substantially the same configurations, and duplicate explanations may be omitted or simplified.

(Embodiment)
[composition]
First, the configuration of the incoming/outgoing management system according to the embodiment will be described. Fig. 1 is a block diagram showing the functional configuration of the incoming/outgoing management system according to the embodiment.

The incoming/outgoing management system 10 shown in FIG. 1 is a system for managing food stored in a refrigerator 20. The incoming/outgoing management system 10 includes a refrigerator 20, an incoming/outgoing management device 30, a server device 40, and an information terminal 50.

Refrigerator 20 is an example of a storage unit, and is installed in a user's home or the like to refrigerate food. FIG. 2 is an external view of refrigerator 20 (and storage/delivery management device 30). As shown in FIG. 2, refrigerator 20 includes a main body 21, a door 22 for accessing a first storage chamber 23 in main body 21, a plurality of shelves 24 provided in first storage chamber 23, and a drawer 25. When door 22 is opened, refrigerator 20 is in a state where food can be placed on each of the plurality of shelves 24 (i.e., food can be stored in first storage chamber 23). When door 22 is opened, refrigerator 20 is in a state where food can be stored in door pocket 22a provided on the inside of door 22. When drawer 25 is opened, refrigerator 20 is in a state where food can be stored in second storage chamber 26 in drawer 25. In the example of FIG. 2, the refrigerator 20 has two doors 22 that cover the first storage compartment 23, but there may be only one door 22 that covers the first storage compartment 23.

The incoming/outgoing management device 30 is attached, for example, to the top surface of the refrigerator 20, and performs information processing to manage food stored in the refrigerator 20. The incoming/outgoing management device 30 is, for example, retrofitted to an existing refrigerator 20, but may also be built into the refrigerator 20. In other words, the incoming/outgoing management device 30 may be a device separate from the refrigerator 20, or may be a device manufactured integrally with the refrigerator 20. As shown in FIG. 1, the incoming/outgoing management device 30 includes an imaging unit 31, a distance measuring unit 32, an information processing unit 33, a memory unit 34, a communication unit 35, and an alarm unit 36.

The image capturing unit 31 captures video from above the refrigerator 20, showing the state when food is being taken in and out of the refrigerator 20. That is, the image capturing unit 31 captures video of an overhead view of the refrigerator 20 with the door 22 open, for example. The video includes as subjects captured the door 22, door pocket 22a, first storage compartment 23, shelf 24, drawer 25, and the floor of the interior space in front of the refrigerator 20. The image capturing unit 31 is realized, for example, by a camera with a fisheye lens, but may also be realized by a camera with a normal lens.

The photographing unit 31 may generate multiple images (still images) by photographing the state when an object is put into or taken out of the refrigerator 20 at a preset time interval, and does not necessarily have to photograph a video. The photographing unit 31 is attached to the housing of the storage/return management device 30, but may also be attached to the refrigerator 20 away from the housing of the storage/return management device 30. In other words, the photographing unit 31 may be retrofitted to the refrigerator 20. In this case, the photographing unit 31 has a magnet, a suction cup, a clip, or the like as a structure that can be retrofitted to the refrigerator 20.

The distance measuring unit 32 is located at the top of the refrigerator 20 and measures the distance from the distance measuring unit 32 to the door 22. In other words, the distance measuring unit 32 measures the degree to which the door 22 is open. The distance measuring unit 32 is realized, for example, by an infrared distance measuring sensor. The distance measuring unit 32 can separately measure the distance from the distance measuring unit 32 to the right door 22 and the distance from the distance measuring unit 32 to the left door 22.

As described below, the distance measurement unit 32 is used to determine whether the door 22 is open or closed. The entry/exit management device 30 may be provided with another sensor for determining whether the door 22 is open or closed, such as an angle sensor, instead of the distance measurement unit 32. The sensor for determining whether the door 22 is open or closed may also be provided by the refrigerator 20, in which case the entry/exit management device 30 does not need to be provided with a sensor for determining whether the door 22 is open or closed.

The information processing unit 33 performs information processing to manage food stored in the refrigerator 20. The information processing unit 33 is realized, for example, by a microcomputer, but may also be realized by a processor or a dedicated circuit. The information processing unit 33 has, as functional components, a determination unit 37, a tracking unit 38, and a control unit 39. The functions of the determination unit 37, the tracking unit 38, and the control unit 39 are realized, for example, by a microcomputer or the like constituting the information processing unit 33 executing a computer program stored in the memory unit 34. The functions of the determination unit 37, the tracking unit 38, and the control unit 39 will be described in detail later.

The memory unit 34 is a storage device that stores the computer program executed by the information processing unit 33 and various information required for the above information processing (such as the setting information and food management information described below). The memory unit 34 also stores a machine learning model for determining (identifying) the type of food. The memory unit 34 is realized, for example, by a semiconductor memory.

The communication unit 35 is a communication circuit that allows the inventory management device 30 to communicate with the server device 40 via the wide area communication network 60. The communication unit 35 is, for example, a wireless communication circuit that performs wireless communication, but may also be a wired communication circuit that performs wired communication. There are no particular limitations on the communication standard for the communication performed by the communication unit 35.

The notification unit 36 issues a notification when it is impossible to determine the type of food being put into or taken out of the refrigerator 20. The notification unit 36 is implemented, for example, by a speaker or buzzer and issues a notification by emitting sound, but may also be implemented by a light-emitting element such as an LED (Light Emitting Diode) and issues a notification by emitting light.

The server device 40 is a computer located outside the facility in which the refrigerator 20 is installed, and specifically, is a cloud server. The server device 40 performs information processing to present information about the food stored in the refrigerator 20. The server device 40 is also an example of a notification device, and performs information processing to notify information about the food stored in the refrigerator 20. The server device 40 includes a communication unit 41, an information processing unit 42, and a memory unit 43.

The communication unit 41 is a communication circuit that enables the server device 40 to communicate with the inventory management device 30 and the information terminal 50 via the wide area communication network 60. The communication unit 41 is, for example, a wired communication circuit that performs wired communication, but may also be a wireless communication circuit that performs wireless communication. There are no particular limitations on the communication standard for the communication performed by the communication unit 41.

The information processing unit 42 performs information processing to present or notify information about food stored in the refrigerator 20. The information processing unit 42 is realized, for example, by a microcomputer, but may also be realized by a processor or a dedicated circuit. The information processing unit 42 has a presentation unit 44 and a notification unit 45 as functional components. The functions of the presentation unit 44 and the notification unit 45 are realized, for example, by a microcomputer or the like constituting the information processing unit 42 executing a computer program stored in the memory unit 43. The functions of the presentation unit 44 and the notification unit 45 will be described in detail later.

The storage unit 43 is a storage device that stores the computer program executed by the information processing unit 42 and various information necessary for the above information processing. The storage unit 43 is realized, for example, by a semiconductor memory.

The information terminal 50 is an information terminal owned by a user. The information terminal 50 is used as a user interface for the user to receive notifications from the server device 40. The information terminal 50 is also used as a user interface when creating training data (described later) for constructing a machine learning model that identifies food types. The information terminal 50 is, for example, a portable information terminal such as a smartphone or a tablet terminal, but may also be a stationary information terminal such as a personal computer. The information terminal 50 includes an input unit 51, a display unit 52, a communication unit 53, an information processing unit 54, a memory unit 55, and an imaging unit 56.

The input unit 51 accepts manual input or voice input from the user. In other words, information is input to the input unit 51 manually or by voice. The input unit 51 is realized by a touch panel, a microphone, or the like.

The display unit 52 displays moving images (or still images) for the user to view. The display unit 52 is realized by a display panel such as a liquid crystal panel or an organic EL (Electro-Luminescence) panel.

The communication unit 53 is a communication circuit that allows the information terminal 50 to communicate with the inventory management device 30 and the information terminal 50 via the wide area communication network 60. The communication unit 53 may also communicate with the inventory management device 30 via a local communication network. The communication unit 53 is, for example, a wireless communication circuit that performs wireless communication, but may also be a wired communication circuit that performs wired communication. There are no particular limitations on the communication standard for the communication performed by the communication unit 53.

The information processing unit 54 performs information processing for creating teacher data. The information processing unit 54 is realized, for example, by a microcomputer, but may also be realized by a processor or a dedicated circuit. The functions of the information processing unit 54 are realized, for example, by the microcomputer or the like constituting the information processing unit 54 executing a computer program stored in the storage unit 55.

The storage unit 55 is a storage device that stores the computer program executed by the information processing unit 54 and various information required for the above information processing. The storage unit 55 is realized, for example, by a semiconductor memory.

The photographing unit 56 is a camera realized by an optical system including an image sensor and a lens.

[Operation example 1]
Next, a management operation of receiving or sending food will be described as a first operation example of the receiving/shipping management system 10. FIG.

First, when the determination unit 37 determines that at least one of the two doors 22 is open based on the measurement result of the distance measurement unit 32 (S11), it activates the photographing unit 31 and starts photographing a video by the photographing unit 31 (S12). FIG. 4 is a diagram showing an example of a video photographed by the photographing unit 31. As shown in FIG. 4, the photographing range of the photographing unit 31 includes the main body 21 (first storage chamber 23), the inside of the two doors 22 (door pockets 22a), and the upper shelf 24a and lower shelf 24b provided in the first storage chamber 23. Note that FIG. 4 also shows a boundary line (dashed line) used to determine whether food is stored or removed from storage. This boundary line is not photographed by the photographing unit 31, but is virtual.

Next, the tracking unit 38 detects food in the video (S13) and tracks the detected food (S14). Various existing algorithms such as R-CNN (Regions with Convolutional Neural Networks) are used to detect food. For example, the tracking unit 38 also detects food in the frame (also referred to as the current frame) following the frame (also referred to as the previous frame) in which the food (also referred to as the tracking target) detected in step S13 is shown. The tracking unit 38 determines, among the foods shown in the current frame, the food that is located closest to the position of the tracking target in the previous frame as the destination of the tracking target. By repeating such processing, the tracking unit 38 can track the food detected in step S13.

The tracking unit 38 may further extract image features of the food detected in step S13 and use the image features as auxiliary information to track the food. The tracking unit 38 may determine the destination of the tracking target based on, for example, the distance of the food detected in the current frame from the tracking target and the similarity of the image features detected in the current frame to the tracking target. This can improve the accuracy of tracking the food.

Furthermore, this method of tracking food is just one example, and other existing algorithms may be used to track food.

Next, the determination unit 37 performs a food entry determination or a food exit determination (S15). The determination unit 37 determines that the food has been exited when the food being tracked in the video moves from inside the refrigerator 20 to outside the refrigerator 20 (to the interior side of the room in which the refrigerator 20 is installed) and crosses the boundary line. The determination unit 37 also determines that the food being tracked in the video moves from outside the refrigerator 20 to inside the refrigerator 20 and crosses the boundary line. The boundary line is determined in advance empirically or experimentally.

Next, the determination unit 37 determines the type of food that has been stored or removed (S16). The determination unit 37 determines the type of food, for example, using a machine learning model. The type of food may be, for example, beverage, vegetable, meat, etc., but may be a more specific type than this. Note that the type of food may be determined at the time of detection of the food in step S13, in which case step S16 can be omitted.

Next, the determination unit 37 updates the food management information stored in the memory unit 34 based on the determination results of steps S15 to S16 (S17). FIG. 5 is a diagram showing an example of food management information. The food management information is information indicating the breakdown of food stored in the refrigerator 20. As shown in FIG. 5, the food management information is, for example, information indicating the type and quantity of food stored in the refrigerator 20. For example, if the determination unit 37 determines that beverages have been removed in steps S15 to S16, the determination unit 37 decreases the quantity of beverages by one in the food management information. Also, if the determination unit 37 determines that vegetables have been received in steps S15 to S16, the determination unit 37 increases the quantity of vegetables by one in the food management information.

Then, when the determination unit 37 determines that both doors 22 are closed based on the measurement results of the distance measurement unit 32 (S18), the operation ends.

As described above, the inventory management system 10 (inventory management device 30) determines whether food is being imported into or exported from the refrigerator 20 (entering food into or exporting food from the refrigerator 20) based on whether the food shown in the video crosses a set boundary line.

If the determination of the type of food fails in step S16, that is, if the type of food cannot be determined, the determination unit 37 may use the notification unit 36 to notify that the type of food cannot be determined. For example, if the notification unit 36 is realized by a speaker, the notification unit 36 outputs a voice message indicating that the type of food cannot be determined. If the notification unit 36 is realized by a light-emitting element, the notification unit 36 notifies that the type of food cannot be determined by emitting light.

In this way, if a notification is given when food type determination fails, the user can take action such as manually correcting the food management information.

[Operation example 2]
Next, a presentation operation of food stored in the refrigerator 20 will be described as a second operation example of the receiving/delivery management system 10. FIG.

The communication unit 35 of the entry/exit management device 30 transmits the food management information to the server device 40 (S21). The communication unit 41 of the server device 40 receives the food management information, and the presentation unit 44 stores the received food management information in the memory unit 43 (S22). The communication unit 35 of the entry/exit management device 30 transmits the food management information to the server device 40, for example, each time a food entry or exit determination is made. Alternatively, the communication unit 35 of the entry/exit management device 30 transmits the food management information to the server device 40 each time an open door 22 is closed. This causes the latest food management information to be stored in the memory unit 43 of the server device 40. The food management information may be transmitted from the entry/exit management device 30 to the server device 40 periodically, such as once at a predetermined time.

On the other hand, when the user performs a specific input to the input unit 51 of the information terminal 50 while away from home, etc., to check the food stored in the refrigerator 20 (contents of the food management information), the input unit 51 accepts such a specific input (S23), and the information processing unit 54 transmits a presentation request to the server device 40 based on the accepted specific input (S24). More specifically, the transmission of the presentation request is performed by the communication unit 53.

The communication unit 41 of the server device 40 receives the presentation request. Based on the received presentation request, the presentation unit 44 transmits presentation information for presenting the contents of the food management information stored in the storage unit 43 to the information terminal 50 (S25). More specifically, the transmission of the presentation information is performed by the communication unit 41.

The communication unit 53 of the information terminal 50 receives the presentation information, and the information processing unit 54 displays a display screen showing the contents of the food management information on the display unit 52 based on the received presentation information (S26). Figure 7 is a diagram showing an example of a display screen showing the contents of the food management information.

In this way, a user of the inventory management system 10 can check the food stored in the refrigerator 20 at home while away from home.

[Operation example 3]
Next, as an operation example 3 of the receiving/shipping management system 10, an operation of creating teacher data for constructing a machine learning model for determining the type of food (in other words, identifying the food) in step S16 of the operation example 1 will be described. Fig. 8 is a sequence diagram of the operation example 3 of the receiving/shipping management system 10. Note that in the following operation example 3, it is assumed that the image capturing unit 31 captures a video with the two doors 22 of the refrigerator 20 open.

The user inputs label information of the food to be photographed by the photographing unit 31 to the input unit 51 of the information terminal 50, and the input unit 51 accepts such input (S31). In other words, the user inputs food label information to the input unit 51. FIG. 9 is a diagram showing an example of an input screen for label information. Label information is input, for example, by selecting from options prepared in advance, but may also be freely input. FIG. 9 shows a display screen when label information is manually input, but label information may also be input by voice uttered by the user.

The label information is, for example, information indicating the type of food (specifically, beverage, vegetable, or meat, etc.), but may also be information indicating a more specific food name (such as milk). Furthermore, the label information may be information indicating both the type and name of the food.

Next, the user inputs shape information indicating the approximate shape of the food to be photographed by the photographing unit 31 to the input unit 51, and the input unit 51 accepts such input (S32). In other words, the user inputs food shape information to the input unit 51. FIG. 9 is also a diagram showing an example of a shape information input screen. Label information is input, for example, by selecting from options prepared in advance, and the user selects the shape that most closely resembles the food. The options include various three-dimensional shapes such as a rectangular prism, a cylinder, and a sphere. FIG. 9 shows the display screen when shape information is manually input, but shape information may also be input by voice uttered by the user.

The shape information is information that is input to display the three-dimensional shape described below, but there may be cases where the three-dimensional shape of the food can be estimated based on the label information input in step S31. For example, if the label information input is "milk carton," it is possible to estimate that the three-dimensional shape is a rectangular prism. In such a case, the control unit 39 may automatically display the three-dimensional shape based on the label information. In other words, manual input of the shape information is not essential and can be omitted.

Next, the information processing unit 54 transmits a display request for the video captured by the image capturing unit 31 to the inventory management device 30 (S33). More specifically, the display request is transmitted by the communication unit 53. The display request is transmitted, for example, in response to an input to select the start image capturing button on the display screen of FIG. 9. The display request includes the label information input in step S31 and the shape information input in step S32.

The communication unit 35 of the inventory management device 30 receives the display request. Based on the received display request, the control unit 39 starts shooting with the shooting unit 31 (S34) and transmits video information for displaying the video shot by the shooting unit 31 to the information terminal 50 (S35). The control unit 39 also transmits guide frame information in addition to the video information (S36). The guide frame information is information for displaying the guide frame and a three-dimensional shape inscribed in the guide frame, which will be described later. More specifically, the transmission of the video information and the guide frame information is performed by the communication unit 35. Note that the transmission of the video information continues during the following steps S35 to S40, during which the video shot by the shooting unit 31 is displayed on the display unit 52 in substantially real time.

The communication unit 53 of the information terminal 50 receives the video information and the guide frame information. The information processing unit 54 displays the video and the guide frame on the display unit 52 based on the received video information and guide frame information (S37). Figure 10A is a diagram showing a display example 1 of the video and the guide frame.

The guide frame is a rectangular frame that instructs the user on where to photograph the food, and in the example of FIG. 10A, it is superimposed on the upper left corner of the video. In other words, the guide frame is a bounding box. In this way, the control unit 39 instructs the user to position the food within the guide frame by displaying the guide frame (sending guide frame information) within the video displayed on the display unit 52.

Also, within the guide frame, a three-dimensional shape inscribed in the guide frame is displayed. This three-dimensional shape is the three-dimensional shape indicated by the shape information included in the display request (the three-dimensional shape input by the user in step S32), and is intended to instruct the user on the posture to take when photographing the food. In other words, the three-dimensional shape indicates the silhouette of the food when photographing it.

At its simplest, the three-dimensional shape is just a frame line, but it may also be a 3D model, illustration, or photograph of the food. When a 3D model, illustration, or photograph is displayed as the three-dimensional shape, the three-dimensional shape may be displayed semi-transparently or flashing to make it easier for the user to overlay the food on the three-dimensional shape.

In this way, the control unit 39 instructs the user on the photographing posture of the food by displaying a three-dimensional shape within the guide frame based on the input shape information (transmitting the guide frame information). The control unit 39 may further instruct the user to point a logo mark or the like of the food toward the photographing unit 31.

The user moves the food in the actual space while looking at the display unit 52 on which a display screen like that shown in FIG. 10 is displayed, and when it is determined that the food is positioned so that it is inscribed in the guide frame and is in a position close to that indicated by the three-dimensional shape, inputs an instruction to take a photograph. The input unit 51 accepts such input (S38). When the input instructing to take a photograph is accepted, the information processing unit 54 transmits photographing trigger information to the inventory management device 30 (S39). More specifically, the transmission of the photographing trigger information is performed by the communication unit 53.

The communication unit 35 of the inventory management device 30 receives the image capture trigger information. Based on the received image capture trigger information, the control unit 39 causes the image capture unit 31 to capture a still image at the timing when the image capture trigger information is received, and stores the captured still image in the memory unit 34 as teacher data by linking it to the label information input in step S31 (S40). In other words, after the guide frame is displayed (after an instruction to position the food within the guide frame is given), the control unit 39 creates teacher data by linking the label information input in step S31 to the still image of the food captured by the image capture unit 31, and stores the created teacher data in the memory unit 34. The teacher data may include information indicating the position of the guide frame (the coordinates of the center position of the guide frame) (i.e., the position information of the location where the food is thought to be captured in the still image).

The processing of steps S36 to S40 is repeated until the number of teacher data (still images) stored in the memory unit 34 reaches a predetermined number, or until the user performs an input indicating the intention to end the creation of teacher data. At this time, the display mode of the guide frame displayed on the display unit 52 is changed (updated) each time guide frame information is transmitted and received in step S36. Figures 10B to 10F are diagrams showing display examples 2 to 6 of the video and guide frame.

The display mode of the guide frame is changed, for example, in the order of Figures 10A to 10F. As shown in Figures 10A to 10F, the display position, shape, size, and aspect ratio of the shape of the guide frame are changed each time guide frame information is transmitted. In other words, the display position, shape, size, and aspect ratio of the shape of the guide frame are changed each time a still image is captured (each time teacher data is created). Note that it is sufficient for at least one of the display position, shape, size, and aspect ratio of the shape of the guide frame to be changed each time guide frame information is transmitted, and it is not necessary that all of the display position, shape, size, and aspect ratio of the shape be changed.

To elaborate on the size of the guide frame, the size of the guide frame corresponds to the distance (vertical distance) from the photographing unit 31 to the food, and a smaller guide frame instructs the user to position the food farther away from the photographing unit 31. Changing the display position of the guide frame has the effect of changing the background image of the food.

When the creation and storage of the training data is completed, the control unit 39 updates the machine learning model (food identification algorithm) by training the machine learning model on the learning data stored in the memory unit 34 (S41).

As described above, the inventory management system 10 has a learning data creation function and also functions as a learning data creation system. The inventory management system 10 instructs the user to position the food within the guide frame by displaying a guide frame within the video displayed on the display unit 52, creates training data by linking the input label information to a still image showing the object captured by the imaging unit 31 after the instruction is given, and changes the display mode of the guide frame each time a still image is captured.

Such an inventory management system 10 can create high-quality training data with appropriate variation in the position and orientation of food within a still image by instructing the user on the position to hold the food and the orientation of the food. Furthermore, with the inventory management system 10, even a user with little knowledge of machine learning models can capture training data (still images) without using special equipment such as a turntable or tripod.

There is no particular limitation on where in the video the guide frame is changed and in what order. Where in the video the guide frame is displayed and in what order may be determined in advance, or may be determined randomly using random numbers, etc. The same applies to the orientation (posture) in which the three-dimensional shape is displayed within the guide frame.

In addition, in step S39, the trigger information was transmitted in response to the user's input instructing the capture of a still image, but it is not essential that the user input the image capture instruction. For example, the information processing unit 54 may detect that the outline of the food has been inscribed within the guide frame by image processing, and may transmit the trigger information in response to such detection. In other words, the capture of a still image may be performed automatically without the need for user input.

[Modification]
In the above-described operation example 1 of the embodiment, the processes from the detection of the food (step S13) to the update of the food management information (step S17) were performed by the inventory management device 30, but some or all of these processes may be performed by the server device 40. For example, the determination of the type of food in step S16 may be performed by the server device 40. In this case, the machine learning model for determining the type of food is stored in the storage unit 43 of the server device 40, and the teacher data created by the inventory management device 30 in the operation example 3 is transmitted from the inventory management device 30 to the server device 40.

In this case, the server device 40 can collect training data from multiple inventory management devices, including the inventory management device 30, and train the machine learning model on a large amount of training data.

In addition, in the operation example 3 of the above embodiment, the information terminal 50 is used as the user interface, but the refrigerator 20 may be used as the user interface instead of the information terminal 50. That is, the refrigerator 20 may be provided with an input unit where the label information and shape information are input by the user, and a display unit where the video captured by the image capturing unit 31 is displayed. Also, one of the input unit 51 and the display unit 52 may be provided by the information terminal 50, and a component corresponding to the other of the input unit 51 and the display unit 52 may be provided by the refrigerator 20. That is, it is not essential that both the input unit 51 and the display unit 52 are provided by the information terminal 50, and it is sufficient that at least one of the input unit 51 and the display unit 52 is provided by the information terminal 50.

In addition, in the above embodiment, the refrigerator 20 may be a refrigerator for general household use, a refrigerator for product display used in retail stores such as convenience stores, or a refrigerator for other commercial use.

In addition, if a single type of food is stored in the refrigerator 20, the type of food does not need to be determined.

Furthermore, refrigerator 20 is one example of a storage unit, and the present disclosure can also be implemented as other storage units that do not have a refrigeration function. Furthermore, the items (in other words, objects) stored in the storage unit are not limited to food, and may be other items. There may also be cases in which items other than food are stored in refrigerator 20.

The teacher data creation system may create teacher data for machine learning models that perform other classifications, in addition to machine learning models that identify foods entering and leaving the refrigerator 20. For example, the teacher data creation system may be realized as a single device equivalent to the information terminal 50, which captures video or still images using the imaging unit 56 provided in the information terminal 50 instead of the imaging unit 31 of the entry/exit management device 30. In this case, the information processing unit 54 of the information terminal 50 includes a control unit 57 (shown by a dashed line in FIG. 1) as a functional component equivalent to the control unit 39.

Such an information terminal 50 functions as a teacher data creation system that creates teacher data used to build a machine learning model that identifies objects. The information terminal 50 includes an input unit 51 to which a user inputs label information of an object, a photographing unit 56 that shoots a video in which the object appears, a display unit 52 that displays the photographed video, and a control unit 57 that instructs the user to position the object within the guide frame by displaying a guide frame within the displayed video. The control unit 57 creates teacher data by linking the input label information to a still image in which the object appears, photographed by the photographing unit 56 after the instruction is given, and changes the display mode of the guide frame each time a still image is photographed. Such information terminals 50 include smartphones, tablet terminals, and personal computers, as well as digital cameras.

Such an information terminal 50 can create training data not only for machine learning models that identify foods entering and leaving the refrigerator 20, but also for machine learning models that perform other types of identification.

[Effects, etc.]
As described above, the incoming/outgoing management system 10 functions as a teacher data creation system that creates teacher data used to build a machine learning model that identifies objects. The incoming/outgoing management system 10 includes an input unit 51 to which a user inputs label information of an object, an image capture unit 31 that captures a video showing the object, a display unit 52 that displays the captured video, and a control unit 39 that displays a guide frame within the displayed video to instruct the user to position the object within the guide frame. The control unit 39 creates teacher data by linking the input label information to a still image showing the object captured by the image capture unit 31 after the instruction is given, and changes the display mode of the guide frame each time a still image is captured.

Such an inventory management system 10 can create training data used to build a machine learning model by specifying the photographing position of an object.

For example, shape information of the object is also input to the input unit 51, and the control unit 39 instructs the user on the shooting posture of the object by displaying a three-dimensional shape within the guide frame based on the input shape information.

Such an inventory management system 10 can create training data used to build a machine learning model by instructing the photographing orientation of an object.

For example, the control unit 39 may display a frame line, a 3D model of an object, an illustration, or a photograph as a three-dimensional shape.

Such an inventory management system 10 can indicate the shooting position of an object using a frame line, a 3D model of the item, an illustration, or a photograph.

Also, for example, the control unit 39 changes the display position of the guide frame each time a still image is captured.

Such an inventory management system 10 can create training data used to build a machine learning model by changing the display position of the guide frame.

For example, the control unit 39 also changes the display position of the guide frame and at least one of the shape and size of the guide frame each time a still image is captured.

Such an inventory management system 10 can create training data used to build a machine learning model by changing the display position of the guide frame and/or the shape and size of the guide frame.

Also, for example, label information is input to the input unit 51 by voice uttered by the user.

Such an inventory management system 10 can accept input of label information based on voice uttered by the user.

For example, at least one of the input unit 51 and the display unit 52 is provided by the information terminal 50. The control unit 39 is provided by a device that communicates with the information terminal 50.

Such an inventory management system 10 can create training data using the information terminal 50 as a user interface.

A teacher data creation method for creating teacher data used to build a machine learning model for identifying an object is executed by a computer such as the inventory management system 10. The teacher data creation method includes a reception step S31 for receiving input of label information of an object from a user, a shooting step S34 for shooting a video showing the object, a display step S37 for displaying the shot video, a control step S37 for instructing the user to position the object within the guide frame by displaying a guide frame within the displayed video, a creation step S40 for creating teacher data by linking the input label information to a still image showing the object that was shot after the instruction was given, and a change step S36 for changing the display mode of the guide frame each time a still image is shot.

This method of creating training data can create training data used to build a machine learning model by specifying the shooting position of an object.

Other Embodiments
Although the embodiments have been described above, the present disclosure is not limited to the above-described embodiments.

For example, in the above embodiment, the guide frame is rectangular, but the guide frame may be a shape other than rectangular, such as circular.

In addition, in the above embodiment, the inventory management system (in other words, the teacher data creation system; the same applies below) is realized by multiple devices. In this case, the components of the inventory management system may be allocated in any way to the multiple devices.

The inventory management system may also be realized by a single device. For example, the inventory management system may be realized as a single device corresponding to the inventory management device of the above embodiment, or as a single device corresponding to a server device, or as a single device corresponding to an information terminal.

Furthermore, the communication method between the devices in the above embodiment is not particularly limited. A relay device (not shown) may be involved in the communication between the devices. Furthermore, the information transmission path described in the above embodiment is not limited to the transmission path shown in the sequence diagram.

In addition, in the above embodiment, the processing performed by a specific processing unit may be executed by another processing unit. Also, the order of multiple processes may be changed, and multiple processes may be executed in parallel.

In addition, in the above embodiment, each component may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or processor reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory.

Furthermore, each component may be realized by hardware. For example, each component may be a circuit (or an integrated circuit). These circuits may form a single circuit as a whole, or each may be a separate circuit. Furthermore, each of these circuits may be a general-purpose circuit, or a dedicated circuit.

In addition, the general or specific aspects of the present disclosure may be realized in a system, an apparatus, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM. In addition, the general or specific aspects of the present disclosure may be realized in any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.

For example, the present disclosure may be realized as a method for creating teacher data executed by a computer, or as a program for causing a computer to execute the method for creating teacher data. The present disclosure may be realized as a computer-readable non-transitory recording medium on which such a program is recorded.

In addition, this disclosure also includes forms obtained by applying various modifications to each embodiment that a person skilled in the art may conceive, or forms realized by arbitrarily combining the components and functions of each embodiment within the scope of the spirit of this disclosure.

10. Inventory management system 20. Refrigerator (storage)
21 Main body 22 Door 22a Door pocket 23 First storage room 24, 24a, 24b Shelf 26 Second storage room 30 Entry/exit management device 31, 56 Photography unit 32 Distance measurement unit 33, 42, 54 Information processing unit 34, 43, 55 Memory unit 35, 41, 53 Communication unit 36 Notification unit 37 Determination unit 38 Tracking unit 39 Control unit 40 Server device 44 Presentation unit 45 Notification unit 50 Information terminal 51 Input unit 52 Display unit 57 Control unit 60 Wide area communication network

Claims (8)

A training data creation system for creating training data used to build a machine learning model for identifying an object, comprising:
an input unit to which label information of the object is input by a user;
An imaging unit that captures a video showing the object;
A display unit on which the captured video is displayed;
a control unit that displays a guide frame within the displayed moving image to instruct the user to position the object within the guide frame,
The control unit is
creating the teacher data by linking the input label information to a still image showing the object captured by the image capturing unit after the instruction is given;
A teacher data creation system which changes a display mode of the guide frame each time the still image is captured. The input unit further receives shape information of the object,
The teaching data creation system according to claim 1 , wherein the control unit instructs the user on the shooting posture of the object by displaying a three-dimensional shape within the guide frame based on input shape information. The teaching data creation system according to claim 2 , wherein the control unit displays a frame line, a 3D model of an object, an illustration, or a photograph as the three-dimensional shape. The teacher data creation system according to claim 1 , wherein the control unit changes a display position of the guide frame each time the still image is captured. The teacher data creation system according to claim 4 , wherein the control unit changes at least one of the display position of the guide frame and the shape, size, and aspect ratio of the shape of the guide frame each time the still image is captured. The teacher data creation system according to claim 1 , wherein the label information is input to the input unit by a voice uttered by the user. At least one of the input unit and the display unit is provided by an information terminal,
The teacher data creation system according to any one of claims 1 to 6, wherein the control unit is provided by a device that communicates with the information terminal. A teacher data creation method executed by a teacher data creation system to create teacher data used to build a machine learning model for identifying an object, comprising:
a receiving step of receiving an input of label information of the object from a user;
A photographing step of photographing a video showing the object;
a display step in which the captured video is displayed;
a control step of displaying a guide frame within the displayed moving image to instruct the user to position the object within the guide frame;
a creating step of creating the teacher data by linking the input label information to a still image showing the object that is captured after the instruction is given;
and a changing step of changing a display mode of the guide frame each time the still image is captured.

Images