JP7250275B2 - Meal identification systems and identification programs - Google Patents

Description

Application of Article 30, Paragraph 2 of the Patent Act (1) Presentation at an exhibition Exhibition name: 2019 Mobak Show (26th International Bakery and Confectionery Related Industry Exhibition) Date: February 20, 2019 to February 23, 2019 Venue: Makuhari Messe

The present invention relates to meal identification systems and identification programs.

Patent document 1 (JP2009-176209A) proposes to settle the meal charge by identifying the tableware on the premise that the unit price is determined if the type of tableware is identified. In this technology, the tableware is identified by imaging the tableware with a stereo camera. Since the unit price is predetermined for each piece of tableware, the charge for the meal can be obtained by adding the unit price corresponding to the identified tableware.

The applicant proposed to identify the type of tableware from the distance image and the color image, limit the types of meal candidates based on the type of tableware, and identify the meal from the color image (Patent Document 2: JP2018-147416A).

By the way, a user of a restaurant, cafeteria, etc., may select and place a plurality of meals such as fried foods and rice dishes on one plate or one tableware. It is difficult to separate a plurality of meals arbitrarily selected by the user and identify images for each meal. In this case, even if the tableware is identified, the meal cannot be identified. Therefore, when the user selects a plurality of types of meals and puts them on one tableware, it is difficult to identify the meals.

JP2009-176209A JP2018-147416A

SUMMARY OF THE INVENTION An object of the present invention is to be able to cope with the case where a plurality of types of meals selected by a user are placed on one tableware in identification of meals.

In the meal identification system of this invention,
a first camera that captures a color image or a black and white image in the tray;
a depth camera that captures depth images in the tray;
a memory for storing a plurality of tableware models comprising a plurality of feature points representing the outline of the tableware, each feature point having data of the direction and depth of the distance from the center of the tableware and the line segment intersecting the center of the tableware;
a scan processing unit that moves the center of the tableware model with respect to the depth image;
The degree of matching between the tableware model and the depth image is obtained by comparing the depth of the feature points of the tableware model and the depth of the corresponding points in the depth image for each feature point at multiple positions during movement by the scanning processing unit. and an evaluation unit that identifies the type of tableware according to the tableware model when the degree of matching satisfies a predetermined condition;
a meal identification unit that selects a meal candidate based on the identified type of tableware and identifies the meal type from the selected meal candidate and the color image or the black and white image;
a monitor with a touch panel that displays the meal identification results;
It has a human interface.

The meal identification program of the present invention uses a color image or black and white image from the first camera, a depth image of tableware from the depth camera, and a monitor with a touch panel to identify the type of meal by a computer.
In the present invention, the computer is
a memory for storing a plurality of tableware models comprising a plurality of feature points representing the outline of the tableware, each feature point having data of the direction and depth of the distance from the center of the tableware and the line segment intersecting the center of the tableware;
a scan processing unit that moves the center of the tableware model with respect to the depth image;
The degree of matching between the tableware model and the depth image is obtained by comparing the depth of the feature points of the tableware model and the depth of the corresponding points in the depth image for each feature point at multiple positions during movement by the scanning processing unit. and an evaluation unit that identifies the type of tableware according to the tableware model when the degree of matching satisfies a predetermined condition;
It operates as a meal identifying section that selects meal candidates based on the identified types of tableware and identifies the meal types from the selected meal candidates and the color image or black and white image.

In this invention, the monitor and the human interface provide
・ Display an add button near the dish to add the number of meals in the dish,
- When the user touches the add button, a menu of meal candidates is displayed based on the identified types of tableware,
• Correct the number and type of meals in the dish based on the type of meal entered by the user from the displayed menu.

In this invention, the degree of matching between the tableware model and the depth image is evaluated while moving the center of the tableware model with respect to the depth image. If the tableware corresponding to the tableware model is at that position, the depth of the feature point and the depth in the depth image match over multiple feature points, otherwise the depth does not match between the tableware model and the depth image. . Since the present invention can perform matching using the remaining feature points even if some of the feature points are hidden, it is suitable for food identification when tableware overlaps and the outline is partially hidden.

By identifying the type of tableware, meal candidates are determined, and meals can be identified using images of the meals. However, accurate identification is difficult when multiple meals are placed on a single tableware. Therefore, an add button for adding the number of meals in the tableware is displayed near the tableware, and when the user touches the add button, a menu of meal candidates is displayed to assist the user's input. Then, based on the type of meal entered by the user, the number and type of meals in the dish are corrected. In this way, even if a single tableware contains a plurality of meals, the type and number of meals can be determined.

Preferably, by the monitor and the human interface,
- display a second add button for adding in-tray and out-of-dish meals;
- when the user touches the second add button, a second menu of meal candidates is displayed;
• Correct the number and type of off-the-shelf meals based on the type of meal entered by the user from the second menu. In this way, the type and number of meals placed outside the dish can also be determined.

Preferably, the tableware model has an attribute that does not require rotation or does not change with 1/n rotation (n is a natural number of 1 or more), and if the attribute of the tableware model does not change with 1/n rotation, the scanning processing unit A rotation control unit is further provided for moving the center of the tableware model and rotating the tableware model by 1/n around the center.

If the shape of the tableware model is not circular, the degree of matching changes depending on the orientation of the tableware model. However, if the tableware model is square, the tableware model returns to its original state when rotated by 90°. Also, if the tableware model is rectangular, the tableware model will return to its original state if it is rotated 180°. Therefore, if the tableware model is given an attribute that does not require rotation (circular) and does not change with 1/n rotation, and the orientation of the tableware is changed according to this attribute, the degree of matching is evaluated even if the orientation of the tableware model and the tableware is different. can.

Preferably, as tableware models, a detailed model having a large number of feature points and provided at least for each type of tableware, and a simple model having a small number of feature points and having a smaller number of models than the number of detailed models, At least two types of models are provided. Here, one simple model is associated with one to a plurality of detailed models. The evaluation unit evaluates the degree of matching with one to a plurality of corresponding detailed models when the degree of matching in the simple model satisfies a predetermined condition.

If the detailed model is applied from the beginning, it takes time to evaluate the degree of matching. On the other hand, if the simple model is applied first and the detailed model is applied when it is promising, the evaluation can be performed at high speed. In particular, when a plurality of detailed models correspond to one simple model, the plurality of detailed models can be evaluated in order when the degree of matching of the simple model is high. This reduces the number of times the simple model is applied and allows faster processing. Of course, there may be a simple model corresponding to only one detailed model.

Preferably, the tableware models are used in order from taller tableware models to shorter tableware models. Tall tableware is difficult to hide and easy to identify. The easily identifiable dishes are identified in order, and the remaining dishes are identified after that.

A block diagram of an example meal identification system. Schematic diagram of an example algorithm Flowchart of an example meal identification algorithm A diagram schematically showing the order of application of models in the example. A diagram schematically showing tableware overlapping each other Illustration showing a circular tableware model A diagram showing a tableware model that is nearly square A diagram showing a tableware model that is nearly rectangular A diagram schematically showing the state in which about half of the types of tableware have been determined Flowchart of an example meal identification algorithm Flowchart of the input algorithm for the type and number of meals in the embodiment A diagram schematically showing the input process of the type and number of meals in the embodiment

A preferred embodiment is shown below. The scope of the present invention should be determined according to the understanding of those skilled in the art based on the description of the claims, taking into consideration the description of the specification and well-known techniques in this field.

Examples are shown in FIGS. 1 to 12. FIG. FIG. 1 shows the configuration of a meal identification system 2, which identifies meals on a tray 02 placed on a top surface 01 of a table. In this specification, a meal means a food or a collection of food that is a unit of identification, such as ``○○ udon'', ``fried chicken'', ``salmon rice ball'', and ``sweet potato tempura''. Meals are assumed to be placed on the tableware W1 to W3 or the like, and to be placed directly on the tray 02, such as rice balls.

The imaging device 3 includes, for example, a color camera 4 and a depth camera 5 . A color image from color camera 4 and a depth image from depth camera 5 are input to meal identification system 2 via input interface 6 , and the color image is stored in memory 7 . Depth images are often accompanied by noise, so the depth is preferably smoothed by the smoothing unit 9 and stored in the memory 8 . If the noise is small, the depth image may be stored in memory 8 without smoothing. Even if the noise is large, the depth image may be stored without smoothing and smoothed each time it is read out.

The sequencer 10 to the evaluation unit 14 assign tableware on the depth image using the tableware model. The interface 6 to a human interface 30 and an interface 36, which will be described later, can be implemented by a CPU of a computer (not shown) and a meal identification program stored in a memory. With respect to tableware models, a simple model is stored in a memory 16 and a detailed model is stored in a memory 17. Once the simple model is determined, one to a plurality of detailed models are determined. The number of detailed models is greater than or equal to the number of types of tableware to be identified, and the number of simple models is less than the number of detailed models, for example, less than the types of tableware. The memory 18 is used for storing work data, storing identification results, and the like.

The sequencer 10 determines the order of simple models to apply, generally starting with taller dish models. Tall tableware models are less likely to hide their contours and are easier to identify. When there are a plurality of tableware models of the same height, the tableware models may be applied in order from the large tableware model that is easy to identify, or may be applied in an arbitrary order without defining rules.

The scan control unit 11 moves the center of the tableware model with respect to the depth image, and scans the depth image with the tableware model. If the tableware model is not circular and the arrangement of the feature points changes due to rotation around the center, the rotation control unit 12 rotates the tableware model around the center at the same time as scanning. For each position of the center of the tableware model (each movement in scanning), the tableware model may be rotated by the full rotation angle within the range of up to one rotation. In order to shorten the scanning time, in simple model matching, each time the center of the tableware moves to the next position, the tableware may be rotated by a predetermined angle.

The evaluation unit 14 calculates the depth and the depth image at each feature point of the tableware model for each position of the tableware model (center position of the tableware model), or for each pair of position and rotation angle when rotating. are compared with the depths of corresponding points in , and their errors are obtained. A corresponding point in the depth image is a point whose position on the image coincides with the feature point of the tableware model. Then, for example, the sum of the squares of the errors is calculated and divided by the total number of feature points in the tableware model to determine the degree of matching. If the corresponding points in the depth image can be estimated to be covered by other tableware, or if the corresponding points in the tableware model are out of the field of view of the depth image, the comparison The number of possible feature points is reduced. In such a case, the above sum of squares may be divided by the number of feature points actually used. The degree of matching is not limited to the sum of squares, and can be evaluated by an appropriate method such as the sum of absolute values of errors.

Upon identifying the type of dishes W1-W3, the meal identifier 20 identifies the type of meal in the dish or the amount in addition to the type. For the meal identification algorithm, for example, the one described in Patent Document 2 is used. The identified meal type is displayed on the touch panel monitor 32 via the human interface 30, and input by the user (the user who brought the tray, the staff at the cafe terrace, etc.) is accepted. This is because it is difficult to identify the meals when multiple meals are served on one tableware, and there are meals such as wrapped rice balls that are placed directly on the tray 02 instead of on the tableware W1 to W3. It's for that. The identification result corrected by the user's input is output to the register device 38 or the like via the output interface 36 .

FIG. 2 shows the overall algorithm of the embodiment, step 1 captures depth and color images of tray 02 . A black-and-white image may be used instead of the color image. Step 2 identifies the type of dish and step 3 identifies the meal in the dish. In step 4, the type of meal is entered via the monitor 32 and the human interface 30 to correct the identification result.

3 and 4 show tableware identification algorithms. The tableware models are scanned in order from the tallest tableware model to the lowest tableware model, and matching is performed from the tableware that is easy to match because it is less likely to be shaded by other tableware. If the tableware model is not circular, it is scanned while being rotated (step 21). The matching does not perform contour extraction from the depth image (cutting the tableware), nor does processing such as estimating the missing region from the depth image (extending to close the contour). Even if some of the feature points are missing from the depth image, they can be identified with the remaining feature points. Experience has shown that identification is successful if more than 50% of the feature points are visible in the depth image. In this way, robustness against lack of feature points is high.

Then, when the simple model matches, it switches to the detailed model. At this time, the center position and orientation of the detailed model are changed within a narrow range to perform accurate matching (step 22). If the matching is successful, the type of tableware is determined, for example, the type of tableware and the position and orientation data are stored in the working data. Preferably, the area covered by the dish is masked so that subsequent identification of the dish does not utilize feature points within this area (step 23). Then, the type, position and orientation of the tableware are output (step 24).

FIG. 5 shows the tableware W1 to W5 on the tray 02, and the outline is the area sandwiched between the solid line and the broken line. The tableware W1 is the tallest, and the feature points are not hidden by other tableware. Tableware W1 to W5 has shapes such as circular (W1, W3 to W5) and square (W2). There are some (W1, W3) where the height decrease is small due to the above. The height of each piece of tableware sharply drops outside the outline.

Examples of tableware models are shown in FIGS. The tableware model consists of a model center C and a plurality of feature points P. The feature points used for both the simple model and the detailed model are indicated by the symbols ●, and symbols such as P1 to P4 are used. A feature point P indicated by a circle and an unnumbered symbol P is used only in the detailed model. The orientation θ of the feature point P is determined, for example, counterclockwise from a line segment S1 extending from the center C to the first feature point P1, and each feature point has a distance r from the center C, an orientation θ, and a depth d. It has Also, the direction from the feature point to the center C is defined as the inner side, and the direction opposite to the center C is defined as the outer side. The depth increases sharply outside the feature points. In order to indicate whether the contour is flat or steep along the direction from the feature point to the center, data such as the change in depth (contour gradient) when moving a certain distance from the feature point to the center is stored at the feature point. may be added to

The tableware model in FIG. 6 is circular and has an attribute that does not require rotation. The tableware model in FIG. 7 has the attribute of 1/4 rotation because it matches the shape before rotation when rotated 1/4, and has the attribute of 1/2 rotation in FIG. If it does not return to its pre-rotation shape without one rotation, the attribute related to rotation is one rotation.

In matching with the tableware model, the center C is moved and the depth image is scanned, and in the case of an attribute that requires rotation, the tableware model is rotated around the center C within a predetermined angle range. For each pair of center position and rotation angle, the depth of the point on the depth image corresponding to the feature point is read, and the error with the depth of the feature point is evaluated. The degree of matching is obtained by accumulating the errors and normalizing them by the number of feature points used.

If the simple model shows a degree of matching equal to or greater than a predetermined value, matching with the detailed model is performed. Note that once the simple model is determined, one to a plurality of detailed models are determined. In addition, compared to the simple model, the detailed model reduces the distance of movement of the center once and the angle of rotation of the center once, thereby performing more accurate matching. However, the range in which the center is moved and the angle range in which the tableware model is rotated are limited to the values obtained by matching the simple model. Then, the tableware model with the highest degree of matching and a predetermined value or more is assumed to be correct.

FIG. 9 shows a state in which matching has been successful up to three out of five pieces of tableware. The height of the tableware model is determined based on the average depth of the feature points. Tall tableware is less likely to hide its outline, so matching with the model is easy. In FIG. 9, matching is started from the tallest platter. In FIG. 9, it is assumed that the tableware on the lower right is matched with the tableware model shown below. By arranging the tableware model, it can be estimated that the corresponding point of the feature point P3 is hidden behind the large plate. Therefore, it is preferable to exclude the feature point P3 from matching. Similarly, any feature points that can be presumed to be out of the field of view of the depth image are preferably excluded from matching.

FIG. 10 shows a meal identification algorithm in a dish. Meal type candidates are registered in the meal identification unit 20 for each type of tableware, and the meal candidates are extracted (step 31). An object exceeding the bottom surface of the tableware model is defined as a meal, and the outline of the meal is extracted from the depth image (step 32). This makes it possible to further limit meal candidates and estimate the amount and number of meals (for example, the number of tempura in one dish). A meal feature is extracted from the depth image and the color image (step 33), and the type of meal is identified from the feature (step 34).

However, meal identification can be error-prone. In particular, when multiple meals are placed on a single tableware, it is difficult to extract individual meals, and it is difficult to obtain an image of the entire meal because the meals overlap each other, which reduces the accuracy of meal identification. do. Therefore, using the algorithm of FIG. 11, the identification result is edited or corrected as shown in FIG. The hardware used is the monitor 32 with a touch panel and the human interface 30 shown in FIG.

As shown in the upper left of FIG. 12, the identification result is displayed on the monitor, for example, the identified meal name is displayed near the image of the meal. A registration button 40 is a button for confirming the identification result, and + buttons 41 to 43 are buttons for adding a meal, which are displayed so as to correspond to tableware. The + button 41 corresponds to the dish W1, and the + button 43 corresponds to the dish W2. The + button may be displayed on all the tableware, and conversely, the + button may not be displayed on some of the tableware. A + button 42 is a button for inputting a meal outside the tableware, such as a rice ball on the upper left of the tray 02 in the figure.

"Pumpkin" (tempura) is displayed in the plate W1, and "Salted Pork Udon" is displayed in the bowl W2, and the identification result of the plate W1 is incorrect. Therefore, the user designates the area that needs to be corrected as indicated by the dashed line in the drawing, touches the + button 41, and inputs an increase in the number of meals (step 41). When correcting the identification result from "pumpkin" to "bait" or the like without adding the number of meals, the meal to be corrected is specified without touching the + button 41 . Also, instead of designating the area or meal to be corrected, it is also possible to simply touch an arbitrary point within the plate W1 to designate correction of the identification result for the plate W1.

In response to the touch of the + button 41, the menu 44 is displayed on the monitor, and the candidate images of the meal in the plate W1 are displayed (step 42). If there are too many menus to be displayed at once, the menu types such as "Tempura" and "Oden" are displayed above the menus, and the user is asked to input which type of menu to display. In the figure, the menu of "tempura" is displayed, and the image and name of each type of tempura are displayed.

The user inputs the type and number of tempura from the menu 44 (step 43), corrects the meal on the plate W1 to, for example, "sweet potato" and "croquette", and touches the enter button 45 to determine the correction result (step 44). ).

Since there is food on the tray 02 that is not placed on the tableware W1, W2 (step 45), the corresponding portion is specified, for example, as indicated by the dashed line in the lower left of FIG. 12, and the + button 42 is touched (step 46). The corresponding part here is the place where the input result is displayed, and does not have to be near the meal to be input.

A menu 46 of meal candidates is displayed on the monitor, and if there are many kinds of menus, the menus are switched for each category such as "rice" and "others", and the user selects "rice" here. Also, images of meals (not shown) are displayed on the menu 46 together with the names of the meals (step 47). The user adds or edits a meal by touching a meal image or meal name on menu 46 (step 48). The types of the entered meals are displayed, for example, "plum rice ball" and "salmon rice ball", and when the input is completed, the user touches the determination button 45 to determine the input content (step 49).

When the input is completed as described above, the meal name is displayed near the image of the meal as shown in the lower right of FIG. , settlement of meal charges, recording of meal contents for each patient, and the like are performed.

01 Top surface of table 02 Tray 2 Tableware identification system 3 Imaging device 4 Color camera 5 Depth camera 6 Input interfaces 7, 8 Memory 9 Smoothing unit 10 Sequencer 11 Scan control unit 12 Rotation control unit 14 Evaluation unit 16-18 Memory 20 Meal identification Unit 32 Monitor 36 Output interface 38 Register device 40 Registration buttons 41 to 43 + buttons 44, 46 Menu display 45 Decision button

C Center of tableware P Characteristic point S1 Line segment θ Orientation W Tableware

Claims (3)

A meal identification system for identifying meal types, comprising:
a first camera that captures a color image or a black and white image in the tray;
a depth camera that captures depth images in the tray;
a memory for storing a plurality of tableware models each comprising a plurality of feature points representing the outline of the tableware, each feature point having data on the distance from the center of the tableware and the orientation and depth of the line segment intersecting the center of the tableware;
a scan processing unit that moves the center of the tableware model with respect to the depth image;
The degree of matching between the tableware model and the depth image is obtained by comparing the depth of the feature points of the tableware model and the depth of the corresponding points in the depth image for each feature point at multiple positions during movement by the scanning processing unit. and an evaluation unit that identifies the type of tableware according to the tableware model when the degree of matching satisfies a predetermined condition;
a meal identification unit that selects a meal candidate based on the identified type of tableware and identifies the meal type from the selected meal candidate and the color image or the black and white image;
a monitor with a touch panel that displays the meal identification results;
with a human interface,
With monitor and human interface,
Display an add button near the dish to add the number of meals in the dish,
when the user touches the add button, displaying a menu of multiple meal candidates based on the identified type of tableware;
A meal identification system that corrects the number and type of meals in a dish based on the type of meal entered by the user from the displayed menu. With the monitor and the human interface,
displaying a second add button for adding in-tray and out-of-dish meals;
When the user touches the second add button, a second menu of meal candidates is displayed;
2. The meal identification system of claim 1, wherein the number and type of off-the-shelf meals are corrected based on the type of meal entered by the user from the second menu. A meal identification program that uses a color image or black and white image from a first camera, a depth image of tableware from a depth camera, and a monitor with a touch panel to identify the type of meal by a computer,
said computer,
a memory for storing a plurality of tableware models comprising a plurality of feature points representing the outline of the tableware, each feature point having data of the direction and depth of the distance from the center of the tableware and the line segment intersecting the center of the tableware;
a scan processing unit that moves the center of the tableware model with respect to the depth image;
The degree of matching between the tableware model and the depth image is obtained by comparing the depth of the feature points of the tableware model and the depth of the corresponding points in the depth image for each feature point at multiple positions during movement by the scanning processing unit. and an evaluation unit that identifies the type of tableware according to the tableware model when the degree of matching satisfies a predetermined condition;
selects a candidate meal based on the type of tableware identified, operates as a meal identification unit that identifies the type of meal from the selected candidate meal and the color image or black and white image;
By said computer,
Displaying on the monitor an add button near the dish for adding the number of meals in the dish, wherein when a user touches the add button, a menu consisting of a plurality of meal candidates based on the identified dish type. is displayed on the monitor,
A meal identification program that corrects the number and type of meals in a dish based on the type of meal entered by the user from a menu displayed on a monitor.

Images