JP7555243B2 - Display device, control device, display method, control method, and program - Google Patents

Description

The present invention relates to a display device, a control device, a display method, a control method, and a program.

Staff at the store that provides the products prepare and serve the products in response to orders from customers. If an order includes multiple products, the staff confirms that all ordered products are ready and then serves the products to the customer. Patent Document 1 discloses a technology that can confirm whether the preparation of the ordered products is complete based on the difference between the standard value of the total weight of the products ordered by the customer and the actual measured value.

JP 2018-055253 A

In general, even products of the same type may vary in weight. In addition, for light-weight products, it may be difficult to determine differences in the type or number of products based on differences in weight. Therefore, it may be difficult to accurately confirm whether a product is ready to be served based on its weight, as in the above-mentioned conventional technology.

The present invention aims to provide a technology that can more accurately confirm that the product to be offered is ready.

A display device according to one aspect of the present invention is a display device that supports the provision of products, and includes a control unit that executes the following: displaying a list of products ordered by a customer based on order identification information included in an image code captured in an imaging area; displaying products included in the product list that have not been provided to the imaging area in a manner different from the products that have been provided to the imaging area based on the results of the imaging; and displaying that the provision of the products included in the product list to the imaging area has been completed.

The present invention provides a technology that can more accurately confirm that the product to be offered is ready.

FIG. 1 is a diagram illustrating a system configuration of an order management system according to an embodiment. FIG. 2 is a sequence diagram showing a processing flow by an order management system according to an embodiment. FIG. 2 is a conceptual diagram illustrating an example of information stored in a management device according to an embodiment. 11 is a conceptual diagram for explaining an example of an image code attached to a product according to one embodiment. FIG. 1 is a conceptual diagram for explaining an example of an image obtained by imaging using an imaging device according to an embodiment. FIG. 2 is a conceptual diagram for explaining an example of a display by a display device according to an embodiment. 1 is a conceptual diagram for explaining an example of an image obtained by imaging using an imaging device according to an embodiment. FIG. 2 is a conceptual diagram for explaining an example of a display by a display device according to an embodiment. 1 is a conceptual diagram for explaining an example of an image obtained by imaging using an imaging device according to an embodiment. FIG. 2 is a conceptual diagram for explaining an example of a display by a display device according to an embodiment. 1 is a conceptual diagram for explaining an example of an image obtained by imaging using an imaging device according to an embodiment. FIG. 2 is a conceptual diagram for explaining an example of a display by a display device according to an embodiment. 1 is a conceptual diagram for explaining an example of an image obtained by imaging using an imaging device according to an embodiment. 1 is a conceptual diagram for explaining an example of an image obtained by imaging using an imaging device according to an embodiment.

An order management system according to one embodiment will be described below. Note that the following embodiment is an example for explaining the present invention, and is not intended to limit the present invention to only this embodiment. Furthermore, the present invention can be modified in various ways without departing from the gist of the invention. Furthermore, a person skilled in the art can adopt an embodiment in which each element described below is replaced with an equivalent, and such an embodiment is also within the scope of the present invention.

The configuration of an order management system according to one embodiment will be described with reference to FIG. 1. The order management system is a system for managing and supporting the acceptance of orders and the provision of ordered products in a store that sells products. The order management system 1 shown in FIG. 1 includes a terminal device 10, a management device 20, an imaging device 30, a display device 40, and a printer 50. The terminal device 10, the imaging device 30, the display device 40, and the printer 50 are communicatively connected to the management device 20. The communication between the devices may be, for example, the Internet, a LAN (Local Area Network), a WAN (Wide Area Network), a dedicated line, a packet communication network, a telephone line, an in-house network, other communication lines, or a combination thereof.

The terminal device 10 is an information processing device for accepting product orders from customers in response to operational input by store staff or customers. The main hardware components of the terminal device 10 include a control unit 11, a communication unit 12, a storage unit 13, an input unit 14, and a display unit 15. The control unit 11 includes a CPU (Central Processing Unit) 11a and a memory 11b. The hardware components will be described later.

The management device 20 is an information processing device for managing the acceptance of product orders from customers and the preparation status of the ordered products. The management device 20 has, as its main hardware components, a control unit 21, a communication unit 22, and a storage unit 23. The control unit 21 has a CPU 21a and a memory 21b. The hardware components will be described later.

The imaging device 30 is a camera that is installed in a store and captures an image of a subject included in an imaging area R, which is a predetermined area. The imaging device 30 is configured, for example, with a lens and an imaging unit, not shown. In this embodiment, the imaging area R is an area in the store where products that have been prepared for serving in response to an order from a customer are temporarily placed. For example, if the product is a dish, the imaging area R is set as an area where the cooked product is temporarily placed after it is carried out from the kitchen. The store staff places the product placed in the imaging area R in a bag, etc., and provides the product to the customer.

The display device 40 is installed in the store and is used to display the preparation status of an ordered item. The display device 40 is installed in a position where the store staff who hands over the ordered item to the customer can check the preparation status of the item. The preparation status of the item includes, for example, "in preparation" and "preparation complete." An example of the display by the display device 40 will be described later. The main hardware components of the display device 40 include a control unit 41, a communication unit 42, a display unit 43, and an input unit 44. These hardware components will be described later.

The printer 50 is a device installed in the store for printing labels to be attached to the ordered products or the packages of those products (hereinafter simply referred to as "products"). Details of the labels will be described later.

Each of the control units 11, 21 controls the operation of each component of the device that includes the control unit. For example, in each of the control units 11, 21, the CPU controls each component of the device and realizes processing by the device by expanding into memory a computer program that includes various instructions stored in a storage unit or the like and executing the program. The control unit 41 is composed of a control circuit or a processor. The control unit 41 controls each component of the display device 40 and realizes display processing by the display device 40 and the like. Details of the processing by each device included in the order management system 1 that is realized by the control of the control units 11, 21, 41 will be described later.

Each of the communication units 12, 22, and 42 is a communication interface that enables the device having the communication unit to communicate with an external device.

Each of the storage units 13 and 23 stores various programs and data necessary for executing processing in the device having the storage unit, various data obtained as a result of the processing, and various data received from an external device.

The input units 14, 44 are devices for user operations and data input. The input units 14, 44 include, for example, button keys, a keyboard, a scanner, and various sensors. The display units 15, 43 are devices for displaying various types of information. The display units 15, 43 are, for example, configured with a liquid crystal display or an organic EL (Electro-Luminescence) display. The input unit 14 and the display unit 15 (or the input unit 44 and the display unit 43) may be configured integrally to form a touch panel.

An example of information processing according to each embodiment will be described below. In the following description, the components shown in FIG. 1 will be referenced as appropriate.

First Embodiment
An example of processing executed by the order management system 1 in this embodiment will be described with reference to Fig. 2. Fig. 2 is a sequence diagram showing an example of processing for managing the contents of orders for products received from customers and the status of preparation for providing the ordered products. This processing is realized by control by a control unit provided in each device in the order management system 1.

The example process shown in Figure 2 shows a process for managing product orders received from two customers (customer A and customer B) and the status of preparation for providing the products.

First, in step S11, the imaging device 30 captures an image of a subject included in a specified imaging area R, and starts transmitting the video data (video image data) obtained by the imaging to the management device 20. As described above, the imaging area R in this embodiment is, for example, an area where products that have been prepared in response to an order from a customer are temporarily placed in order to be offered to the customer. The management device 20 sequentially performs an analysis process on the received video data.

Next, in step S12, the terminal device 10 transmits information on the order contents of customer A (order information) input via the input unit 14 to the management device 20. In step S13, the management device 20 registers the order by storing the order information of customer A received from the terminal device 10 in the memory unit 23.

Next, in step S14, the terminal device 10 transmits information on the order details of customer B input via the input unit 14 to the management device 20. In step S15, the management device 20 registers the order by storing the order information of customer B received from the terminal device 10 in the memory unit 23.

With reference to FIG. 3, an example of order information stored in the storage unit 23 in steps S13 and S15 of FIG. 2 will be described. As shown in FIG. 3, the order information stored in the storage unit 23 includes the following items: "customer number", "serial number", and "order details". "Customer number" is information for identifying an order by a customer (order identification information). "Serial number" is information for identifying products included in an order (product identification information), and is assigned to the products with a symbol according to the number of products included in an order. In the example shown in FIG. 3, "serial number" is a consecutive number assigned to each product included in an order. Therefore, if an order includes four products, each product is assigned a number from 1 to 4. As another example, instead of "serial number", other symbols such as alphabets may be used to identify products. "Order details" is the name of the product included in the order.

In the example shown in FIG. 3, the "customer number" (i.e., order identification information) "166" of customer A's order is associated with the "order details" of "regular beef bowl with cheese," "salad," "Tabasco," and "Japanese dressing," and each "order detail" is associated with the "sequential numbers" "1," "2," "3," and "4" which are stored in association with each "order detail." In other words, customer A's order includes four items. Similarly, in the example shown in FIG. 3, the "customer number" "174" of customer B's order is associated with the "order details" of "regular beef bowl" and "fried chicken," and each "order detail" is associated with the "sequential numbers" "1" and "2" which are stored in association with each "order detail." In other words, customer B's order includes two items.

Returning to the explanation of FIG. 2, in step S16, the management device 20 transmits information on the products included in the order registered in steps S13 and S15 to the printer 50. In step S17, the printer 50 prints out a label to be attached to the product that is ready to be served based on the product information received from the management device 20. The label is, for example, a sticker made of paper or plastic, with a printed image code (for example, a QR code (registered trademark) or a barcode) printed on one side and glue applied to the other side. The image code is generated, for example, by encoding information identifying the order and information on one product included in the order. For example, if one order includes four products, four labels on which an encoded image code containing information on one of the four products is printed are output from the printer 50. As a variant, in step S17, the printer 50 may print out a sheet on which the image code is printed (for example, a paper order slip on which the image code and characters indicating the ordered product information are printed) instead of (or in addition to) a label attached to the product. For example, if one order includes four products, the printer 50 outputs one sheet on which an encoded image code containing information about the four products is printed.

In this embodiment, the image code displayed on the label printed out from the printer 50 is generated by encoding the minimum necessary information so that the size of the image code is small. In this embodiment, the image code displayed on the label may be generated, for example, by encoding information consisting of the customer number described with reference to FIG. 3 and the sequential number of the product included in the order identified by the customer number. As a variation, instead of the sequential number, other symbols such as letters of the alphabet may be used to identify the products in an order. The label is attached by the store staff to the product that is ready to be served so that the label contents correspond to the product information.

With reference to FIG. 4, the difference between the label in this embodiment attached to the product and a label in another form (hereinafter referred to as the "label of the comparative example") will be described. FIG. 4 shows the label L1 in this embodiment and the label L12 of the comparative example. The label L1 displays the image code c1. The label L12 displays the image code c12. Furthermore, each label displays the product name ("Regular beef bowl cheese"), the expiration date, and the processing time. The image code c1 is generated by encoding information consisting of, for example, a customer number and a serial number of the product as described above. Here, in this embodiment, the serial number of the product is information that can identify a product within one order (rather than information that can identify a product among all the products handled in the store), so it can be made to have a small data size (for example, 1 byte). In contrast, the image code c12 displayed on the label L12 of the comparative example is generated by encoding information that includes at least the customer number (or order identification information) and information that can identify a specific product among all the products handled in the store. In this case, the information that can identify a specific product from all the products handled in the store will have a larger data size than the sequential number in the image code c1. For at least this reason, as shown in FIG. 4, the image code c1 in this embodiment can be generated in a smaller size than the image code c12. Note that this does not exclude the possibility of replacing the label in this embodiment with a label of a comparative example.

In general, it can be difficult to attach a label displaying an image code to a product due to factors such as the small size of the product. By reducing the size of the image code as in this embodiment, the size of the label can also be reduced, making it possible to attach labels to more products.

Returning to the explanation of FIG. 2, when a product that has been prepared for serving according to customer A's order is placed in the imaging area R by a store staff member, in step S18, the management device 20 detects the image code attached to the product as a result of an analysis process of the video data received from the imaging device 30. The management device 20 further analyzes the image code to detect the customer number and sequential number. For example, the management device 20 detects the customer number "166" and the sequential number "1".

With reference to FIG. 5, an example of video data in which an image code was detected in step S18 in FIG. 2 will be described. FIG. 4 shows an image a1 obtained by imaging the imaging area R from above in the vertical direction using the imaging device 30. In image a1, the imaging area R is divided into sub-areas r1 and r2. Products included in the same order are placed in one sub-area. Thus, a staff member places products included in one order in sub-area r1, and products included in another order in sub-area r2. Each sub-area is defined by any method. For example, the sub-areas may be defined by dividing the imaging area R into two, left and right (or top and bottom). Alternatively, two or more plates may be placed in the imaging area R, with each plate defining one sub-area.

In the example shown in FIG. 5, the image of sub-area r1 includes an image of product P1. The image of product P1 also includes an image of label L1 attached to the package of product P1. Label L1 includes image code c1. Image code c1 is a code generated by encoding information including the customer number and sequential number described with reference to FIG. 3 above.

Note that, if a sheet (e.g., an order slip) on which an image code is printed as described as a modified example in step S17 is printed out, the image code printed on the sheet may be detected in step S18 instead of (or together with) the image code attached to the product. In this case, for example, the sheet placed in the imaging area R by a store staff member is first imaged by the imaging device 30. The management device 20 may detect the image code attached to the sheet as a result of an analysis process of the video data including the image of the sheet received from the imaging device 30, and may analyze the image code to detect the customer number and the serial number. In the processes in this embodiment and other embodiments described below, the image code printed on the sheet may be detected instead of (or together with) the image code attached to the product, and the subsequent processes may be performed.

Returning to the explanation of FIG. 2, in step S19, the management device 20 refers to the order information stored in the memory unit 23, and acquires a list of order details associated with the customer number detected in step S18, and information on the order details associated with the serial number detected in step S18. The management device 20 transmits the acquired information to the display device 40 together with the customer number. For example, the management device 20 acquires a list of four products (order details) associated with customer number "166", and information on the product (order details) "regular beef bowl with cheese" associated with serial number "1". The management device 20 then transmits the acquired information and customer number "166" to the display device 40.

In step S20, the display device 40 displays a list of products ordered by the customer and the preparation status of product provision on the display unit 43 based on the information received from the management device 20. In detail, the display device 40 displays the customer number detected in step S18 and a list of order details associated with the customer number on the display unit 43. Furthermore, the display device 40 displays, in the list of order details, the order details associated with the serial number detected in step S18 in a different manner from other order details.

With reference to FIG. 6, an example of an image displayed on the display unit 43 in step S20 of FIG. 2 will be described. This example is based on the assumption that order information such as that shown in FIG. 3 is stored in the memory unit 23 of the management device 20, and that customer number "166" and serial number "1" were detected in step S18 of FIG. 2.

6, the image displayed on the display unit 43 indicates that the current preparation status of the order with customer number "166" is "preparing". In addition, "Regular beef bowl with cheese", "Salad", "Tabasco", and "Japanese dressing" are displayed as a list of the contents (product information) of the order with customer number "166". The products included in the list of order contents include not only the products explicitly ordered by the customer, but also products that accompany the products explicitly ordered by the customer (for example, "Japanese pepper" or other toppings that accompany the explicitly ordered product "eel bowl"). The number of products ordered and the number of products that are ready to be served are displayed to the right of each product information. For example, "0/1" is displayed to the right of "salad", which indicates that the number of salads ordered is 1 and the number of products that are ready to be served is 0. If the number of salads ordered is 2 and the number of products that are ready to be served is 1, "1/2" is displayed. Additionally, a check mark is displayed to the left of product information that is ready to be served, indicating that the product is ready to be served. As a variation, the display unit 43 may display product information that is ready to be served in the list of order details in a color different from other product information.

As described above, according to this embodiment, products included in the product list that are not provided in the imaging area R are displayed on the display device 40 in a manner different from the products provided in the imaging area R, based on image data obtained by the imaging device 30 as an imaging result. In addition, the products provided in the imaging area R are identified, for example, based on product information included in the image code of the product imaged in the imaging area R.

Returning to the explanation of FIG. 2, if another product included in customer A's order is provided in the imaging area R after step S20, in step S21, the management device 20 detects the image code attached to the product as a result of the analysis process of the video data received from the imaging device 30. The management device 20 further detects the customer number and the serial number by analyzing the image code. For example, the management device 20 detects the customer number "166" and the serial number "2". The management device 20 further refers to the memory unit 23 and acquires information on the order contents associated with the detected customer number and serial number. For example, the management device 20 acquires information on the product contents "salad" associated with the customer number "166" and the serial number "2".

Referring to FIG. 7, an example of video data in which an image code was detected in step S21 of FIG. 2 will be described. FIG. 7 shows image a2 obtained by imaging imaging area R from vertically above using imaging device 30. Compared to image a1 of FIG. 5, sub-area r1 of image a2 includes an image of product P2 in addition to an image of product P1. Furthermore, the image of product P2 includes an image of label L2 attached to the package of product P2. Label L2 includes image code c2.

Next, in step S22 of FIG. 2, the management device 20 transmits the order content information acquired in step S21 to the display device 40. In step S23, the display device 40 updates the display on the display unit 43 based on the information received from the management device 20.

With reference to FIG. 8, an example of an image displayed on the display unit 43 in step S23 of FIG. 2 will be described. This example is based on the assumption that order information such as that shown in FIG. 3 is stored in the memory unit 23 of the management device 20, and that customer number "166" and serial number "2" were detected in step S21 of FIG. 2.

As shown in FIG. 8, the image displayed on the display unit 43, compared to the image shown in FIG. 6, displays "1/1" to the right of "salad" and also displays a check mark to the left of "salad." In other words, it is shown that, of the items included in the list of ordered items, salad was also provided in the imaging area R.

2, the remaining products included in customer A's order are provided to sub-area r1 of image capture area R, and one product included in customer B's order is provided to sub-area r2. At this time, in step S24, the management device 20 detects the customer number, serial number, and product information (e.g., product name) provided to image capture area R based on the results of the analysis process of the video data received from the image capture device 30.

Referring to FIG. 9, an example of video data in which the customer number, serial number, and product information were detected in step S24 in FIG. 2 will be described. FIG. 9 shows image a3 obtained by imaging imaging area R from vertically above using imaging device 30. Compared to image a2 in FIG. 7, sub-area r1 of image a3 further includes images of products P3 and P4 in addition to images of products P1 and P2. Also, sub-area r2 of image a3 includes an image of product P5.

The products P3 and P4 are not labeled. In this case, the management device 20 identifies the products provided in the sub-area r1 of the imaging area R based on the image recognition processing of the products for the sub-area r1 of the image a3. Any technology such as artificial intelligence can be adopted for the image recognition processing. For example, a trained model is constructed by machine learning or deep learning with the image of the product as an explanatory variable and the product information (e.g., product name) as an objective variable, and the data of the model is stored in the storage unit 23 of the management device 20 or in the storage unit of another device. By using the trained model constructed in this way, for example, the product information "Tabasco" of the product P3 can be obtained as an output by using the image of the product P3 as an input. Also, for example, the product information "Japanese dressing" of the product P4 can be obtained as an output by using the image of the product P4 as an input. That is, in this example, the management device 20 identifies that "Tabasco" and "Japanese dressing", which are products (order contents) included in the order of customer A, have been provided in the sub-area r1 of the imaging area R based on the image a3. The start of image recognition processing (e.g., processing employing artificial intelligence, etc.) for a product (e.g., product P3) can be set to any timing, but the start of image recognition processing may also be triggered by the detection of an image code displayed on the label of another product (e.g., product P1) (or an image code printed on the sheet).

In addition, the image of product P5 in sub-area r2 in FIG. 9 includes an image of label L5 attached to the package of product P5. Label L5 includes image code c5. Management device 20 detects the customer number and the serial number by analyzing image code c5. Next, management device 20 refers to the order information stored in memory 23, and acquires list information of order contents associated with the detected customer number and information of order contents associated with the detected serial number. For example, management device 20 detects customer number "174" and serial number "1". In this case, management device 20 refers to memory 23, and acquires list information of two products (order contents) associated with customer number "174", which is customer B's order identification information, and information of the product (order contents) associated with serial number "1", "regular beef bowl" (see FIG. 3).

Returning to the explanation of FIG. 2, in step S25, the management device 20 transmits to the display device 40 the list information of the order details of customer B acquired in step S24, information on the products provided in the imaging area R among the orders of customers A and B, and the customer number.

In step S26, the display device 40 updates the display on the display unit 43 based on the information received from the management device 20 in step S25.

With reference to FIG. 10, an example of an image displayed on the display unit 43 in step S26 of FIG. 2 will be described. This example assumes that order information such as that shown in FIG. 3 is stored in the memory unit 23 of the management device 20, and that "Tabasco" and "Japanese dressing" are detected in sub-area r1, and customer number "166" and serial number "1" are detected in sub-area r2 in step S25 of FIG. 2.

As shown in FIG. 10, the image displayed on the display unit 43 includes areas 101 and 102. Area 101 displays a list of products included in customer A's order (customer number "166") and the preparation status of the products. Area 102 displays a list of products included in customer B's order (customer number "174") and the preparation status of the products. Compared to the image shown in FIG. 8, the image in area 101 displays "1/1" to the right of "Tabasco" and "Japanese dressing" and a check mark is displayed on the left. In other words, area 101 indicates that the four products included in the list of ordered products have been provided to the imaging area R. Therefore, "Provided OK" is displayed at the top of area 101 to indicate that the preparation of the ordered products has been completed. When the ordered product has been provided to the imaging area R, the display device 40 may announce by voice from a speaker (not shown) that the ordered product is ready to be provided, together with (or instead of) displaying "Provided OK."

The image in area 102 shows that the current preparation status of the order for customer number "174" is "in preparation." In addition, a list of the contents (product information) of the order for customer number "166" is displayed, including "regular beef bowl" and "fried chicken." To the right of each product information, the number of products ordered and the number of products that are ready to be served are displayed. In addition, a check mark is displayed to the left of product information that is ready to be served. For example, "1/1" is displayed to the right of "regular beef bowl," and a check mark is displayed to the left. In other words, in this example, it is shown that preparation for serving the regular beef bowl is complete.

In this way, according to the example shown in FIG. 10, the control unit 41 of the display device 40 controls the display unit 43 to display a list of product information identified based on the customer number of the order by customer A in a first area of the display unit 43, and to display a list of product information identified based on the customer number of the order by customer B in a second area of the display unit 43.

As described above, according to this embodiment, the display device 40 displays a list of products ordered by a customer based on the customer number included in the image code of the product captured in the imaging area R, and based on the results of imaging by the imaging device 30, displays products included in the product list that have not been provided to the imaging area R in a manner different from the products that have been provided to the imaging area R. Furthermore, the display device 40 displays that the provision of the products included in the product list to the imaging area R has been completed. As a result, the store staff can more accurately confirm that the preparation for providing the products has been completed by preparing the ordered products for provision while checking the display on the display device 40.

In addition, in this embodiment, if the product provided in the imaging area R is not included in the product list displayed on the display device 40, the display device 40 (control unit 41) may display that the product provided in the imaging area R is not included in the order. In detail, for example, the management device 20 identifies a product provided in sub-area r1 of the imaging area R in order to manage the preparation status of customer A's order, and determines that the identified product is not a product included in customer A's order. At this time, the display device 40 is controlled to display that the product provided in sub-area r1 is not included in customer A's order.

In addition, the computer program for executing the processing in this embodiment may be installed or loaded into the computer through various recording media such as optical disks, magnetic disks, and semiconductor memories, or by downloading via a communication network, etc.

[Modification]
For example, there are products for which it is difficult to attach a label to the product and also difficult to identify the product by image recognition processing. For such products, the display device 40 may indicate that the product has been provided in the image capture area R in response to an operation on the display device 40. For example, it is assumed that it is difficult to attach a label to the product, miso soup, and also difficult to identify the miso soup by image recognition processing. In this case, if miso soup is included in the order of customer A and the miso soup is provided in the image capture area R, the display device 43 may indicate that the miso soup has been provided in the image capture area R in response to a store staff member tapping, for example, on a portion of the list of order details displayed on the display unit 43 of the display device 40 where information about the miso soup is displayed.

As another variation, even if a product not ordered by a customer is provided in the imaging area R, the management device 20 may be controlled to ignore the detection result of the product. For example, even if a product not ordered by a customer is provided in the imaging area R, the management device 20 may be controlled not to display an error message on the display unit 43. Also, even if a product not ordered by a customer is provided in the imaging area R, the management device 20 may be controlled to display on the display unit 43 that preparation for provision is complete when all ordered products have been provided in the imaging area R.

Second Embodiment
This embodiment is an embodiment for enhancing the convenience of the order management system 1. In this embodiment, the management device 20 stores information on a list of products included in an order by a customer and information on the preparation status of product provision in the storage unit 23 based on the imaging results in the imaging area R. Even if a product provided in the imaging area R is temporarily undetectable due to some obstruction (e.g., a human body) after it has been detected, the preparation status of product provision is managed based on the information stored in the storage unit 23.

The contents of this embodiment described below can be combined with part or all of each of the other embodiments as appropriate. In this embodiment, the same components as those in the other embodiments are appropriately designated by the same reference numerals, and the description thereof will be omitted or simplified.

In this embodiment, the management device 20 stores in the storage unit 23 information on the ordered items identified based on the order identification information included in the image code of the item captured in the imaging area R, and information on the item provided in the imaging area R identified based on the result of the imaging by the imaging device 30. For example, as shown in FIG. 2, the management device 20 identifies information on the ordered items based on the customer number (order identification information) included in the image code of the item captured in the imaging area R (e.g., step S18), and further identifies information on the item provided in the imaging area R based on the result of the imaging of the item (e.g., based on the image code and image recognition processing) (e.g., steps S18, S21, and S24, etc.). The management device 20 stores the identified information in the storage unit 23. That is, the management device 20 stores information on a list of items included in the customer's order and information on the preparation status of the item provision in the storage unit 23 based on the result of the imaging in the imaging area R. Examples of information on a list of items included in the customer's order and information on the preparation status of the item provision are shown in, for example, FIG. 8, etc.

The management device 20 also controls the display device 40 to display a list of ordered products (e.g., step S20 or S23 in FIG. 2). At this time, among the products included in the product list, products that have not been provided in the imaging area R are displayed on the display device 40 in a manner different from the products that have been provided in the imaging area R. For example, as shown in FIG. 8, among the ordered products, products that have been provided in the imaging area R are displayed in a highlighted manner, different from the other products (e.g., a check mark is displayed).

After that, when, for some reason, all of the detected products (e.g., all of the products ordered by customer A) can no longer be detected, the management device 20 controls the display device 40 to delete the product list from the display. For example, the image as shown in FIG. 8 displayed on the display unit 43 of the display device 40 becomes invisible.

After that, when the ordered items that were no longer identified in the imaging area R are identified again (when at least a portion of the ordered items are identified again), the management device 20 controls the display device 40 to display the list of ordered items again based on the information stored in the memory unit 23. For example, an image such as that shown in FIG. 8 is displayed again on the display unit 43 of the display device 40.

Then, the management device 20 controls the display device 40 to display a message indicating that the products included in the product list have been provided to the imaging area R. For example, the management device 20 controls the display device 40 to display an image such as that shown in area 101 of FIG. 10.

As described above, according to this embodiment, the storage unit 23 of the management device 20 stores information on the ordered items identified based on the order identification information included in the image code of the item captured in the imaging area R, and information on the item provided in the imaging area R identified based on the result of the above-mentioned imaging. The management device 20 causes the display device 40 to display a list of ordered items based on the information stored in the storage unit 23, and causes the display device 40 to display items included in the list of ordered items that have not been provided in the imaging area R in a form different from that of the items provided in the imaging area R based on the result of the above-mentioned imaging. When none of the ordered items are identified in the imaging area R based on the result of the above-mentioned imaging, the management device 20 controls the display device 40 to delete the list of ordered items from the display. Thereafter, when an ordered item that is no longer identified in the imaging area R is identified again, the management device 20 controls the display device 40 to display a list of ordered items based on the information stored in the storage unit 23.
That is, according to this embodiment, even if a product provided in the imaging area R is detected and then becomes undetectable due to some obstruction (e.g., a person's body), the management device 20 determines that the product has been provided to the customer and does not reset the information on the preparation status for providing the product, but keeps it stored in the storage unit 23. Thereafter, when an ordered product that was no longer identified in the imaging area R is identified again (when at least a part of the ordered product is identified again), the management device 20 controls the display device 40 to display again a list of the ordered products based on the information stored in the storage unit 23. As a result, according to this embodiment, the preparation status for providing the product can be properly managed.

Third Embodiment
Similar to the second embodiment, this embodiment is an embodiment for improving the convenience of the order management system 1. The second embodiment is an embodiment for the case where all of the detected products (e.g., all of the products ordered by customer A) are no longer detected, whereas this embodiment is an embodiment for the case where some of the detected products are no longer detected.

The contents of this embodiment described below can be combined with part or all of each of the other embodiments as appropriate. In this embodiment, the same components as those in the other embodiments are appropriately designated by the same reference numerals, and the description thereof will be omitted or simplified.

In this embodiment, similar to the second embodiment, the management device 20 stores in the memory unit 23 information on the ordered items identified based on the order identification information included in the image code of the item captured in the imaging area R, and information on the item provided in the imaging area R identified based on the result of imaging by the imaging device 30. Also, in this embodiment, the management device 20 controls the display device 40 to display, based on the result of imaging the imaging area R by the imaging device 30, items included in the list of ordered items that are no longer identified in the imaging area R in the same format as the items provided in the imaging area.

For example, as described above, in step S23 of FIG. 2, the management device 20 controls the display device 40 to display an image as shown in FIG. 8. In FIG. 8, among the products included in the list of ordered products, products that have not been provided in the imaging area R are displayed in a manner different from the products that have been provided in the imaging area R. In such a case, it is assumed that some of the products included in the list of ordered products can no longer be identified in the imaging area R.

Figure 11 shows an example of a case where some of the products included in the list of ordered products in the video data shown in Figure 7 are no longer identified in the imaging area R. In Figure 11, product P1 shown in Figure 7 is hidden by an obstruction M1. That is, in the example shown in Figure 11, product P1 (or image code c1 of product P1) is no longer identified in the imaging area R. Even in such a case, according to this embodiment, the management device 20 controls the display device 40 to display the products that are no longer identified in the imaging area R in the same form as the products provided in the imaging area, based on the information stored in the memory unit 23. For example, the display device 40 maintains the display of an image as shown in Figure 8.

Furthermore, when new products are identified in the imaging area R based on the results of imaging by the imaging device 30, the management device 20 can control the display device 40 to change the number of products provided in the imaging area R. In other words, it is possible to add additional products without resetting the number of products already counted.

For example, suppose that customer A orders two "regular beef bowl with cheese" products, and the first "regular beef bowl with cheese" is provided in imaging area R, causing "regular beef bowl with cheese 1/2" to be displayed on display device 40 as shown in FIG. 12. According to this embodiment, even if a second "regular beef bowl with cheese" is then stacked on top of the first "regular beef bowl with cheese", hiding the first "regular beef bowl with cheese", it is still recognized that the first "regular beef bowl with cheese" is ready to be served, and that the second "regular beef bowl with cheese" is ready to be served. In other words, the display "regular beef bowl with cheese 1/2" in FIG. 12 is changed to "regular beef bowl with cheese 2/2".

As described above, according to this embodiment, the storage unit 23 of the management device 20 stores information on the ordered items identified based on the order identification information included in the image code of the item captured in the imaging area R, and information on the item provided in the imaging area R identified based on the result of the above imaging. The management device 20 displays a list of ordered items on the display device 40 based on the information stored in the storage unit 23, and displays items included in the list of ordered items that have not been provided in the imaging area R on the display device 40 in a different form from the items provided in the imaging area R based on the result of the above imaging. The management device 20 displays items included in the list of ordered items that are no longer identified in the imaging area R on the display device 40 in the same form as the items provided in the imaging area R based on the result of the above imaging. After that, the management device 20 changes the number of items provided in the imaging area R displayed on the display device 40 in response to a new item being identified in the imaging area R based on the result of the above imaging.
That is, according to this embodiment, even if a product provided in the imaging area R is detected and then becomes undetectable due to some obstruction (e.g., a person's body or another ordered product), the management device 20 does not determine that the product is being prepared for provision, and controls the display device 40 to maintain its display. Furthermore, in response to a new product being identified, the management device 20 changes the number of products provided in the imaging area R displayed on the display device 40. As a result, according to this embodiment, the preparation status for providing products can be properly managed.
When only one item is ordered and that item is no longer identified based on the result of imaging by the imaging device 30, the management device 20 may delete the ordered item from the display of the display device 40, and thereafter, when the ordered item that is no longer identified in the imaging area R is identified again, the management device 20 may control the display device 40 to display the ordered item based on the information stored in the storage unit 23. However, the present invention is not limited to this method, and in the above case, the management device 20 may display the item that is no longer identified in the imaging area R on the display device 40 in the same form as the item provided in the imaging area R.

Fourth Embodiment
In this embodiment, it is determined whether or not the product provided in the imaging area R is correct based on the product identified based on the image code and the image recognition result for the product image.

The contents of this embodiment described below can be combined with part or all of each of the other embodiments as appropriate. In this embodiment, the same components as those in the other embodiments are appropriately designated by the same reference numerals, and the description thereof will be omitted or simplified.

In this embodiment, the management device 20 identifies an object that fits within a rectangle in the imaging area R. In detail, the management device 20 identifies an object (product) that fits within a rectangle, which is a virtually defined shape, while changing the position and size of the rectangle in the image of the imaging area R obtained by the imaging device 30. The virtually defined shape is not limited to a rectangle, and may be any other predetermined shape, such as another polygon, ellipse, circle, or irregular shape.

An example of an object that fits within a rectangle will be described with reference to FIG. 13. As shown in FIG. 13, a rectangle N6 is virtually defined in the imaging area R. FIG. 13 also shows an object (product P6) that fits within the rectangle N6, which is identified by changing the position and size of the rectangle N6.

The management device 20 then compares the ordered product identified based on the order identification information contained in the image code within the rectangle with the product identified based on the image recognition results for the object within the rectangle.

In the example shown in FIG. 13, the management device 20 compares the ordered product identified based on the customer number (order identification information) included in the image code c6 in the rectangle N6 with the product identified based on the image recognition results for the object in the rectangle N6. The method of identifying the ordered product based on the customer number is as described in the first embodiment. Any image recognition process can be used to identify the product based on the image recognition results. For example, a trained model is constructed by machine learning or deep learning using the product image as an explanatory variable and the product information (e.g., product name) as an objective variable, and the data of the model is stored in the storage unit 23 of the management device 20 or in a storage unit of another device. By using the trained model constructed in this way, it is possible to obtain, for example, product information of the product P6, "regular beef bowl with cheese", as an output, using the image of the product P6 as an input.

Based on the result of the comparison, the management device 20 determines whether the product provided in the imaging area R is correct. In other words, if the ordered product identified based on the order identification information corresponds to the product identified based on the image recognition results for the object within the rectangle (within the specified shape), the provided product is determined to be correct, and if they do not correspond, the provided product is determined to be incorrect.

For example, if the ordered item identified based on the customer number included in image code c6 in rectangle N6 is "regular beef bowl with cheese" and the item identified based on the image recognition results for the object in rectangle N6 is "regular beef bowl with cheese", the management device 20 determines that the item provided is correct. On the other hand, if the ordered item identified based on the customer number is "regular beef bowl with cheese" and the item identified based on the image recognition results for the object in rectangle N6 is "regular beef bowl" (i.e., if the image recognition process does not recognize a cheese topping on the beef bowl), the management device 20 determines that the item provided is incorrect.

As described above, according to this embodiment, the product (e.g., the type of product and toppings placed on the product) is identified by image recognition processing on the image of the product (object) provided in the imaging area R. Furthermore, based on the comparison result between the ordered product identified based on the order identification information and the product identified by the image recognition processing, it is determined whether or not the product provided in the imaging area R is correct. As a result, it is possible to prevent mistakes in providing products to customers.

Also, as shown in FIG. 14, when identifying an image code using the above rectangle (predetermined shape), multiple image codes may be identified within one rectangle. FIG. 14 shows products P6 and P7 provided in the imaging area R. Product P6 is contained within rectangle N6, and product P7 is contained within rectangle N7. Furthermore, in addition to image code c7 attached to product P7, image code c7 is contained within rectangle N7. That is, multiple image codes are identified within rectangle N7. In such a case, the management device 20 identifies the ordered product based on the order identification information contained in the image code closest to the center of rectangle N7 (i.e., image code c7). The management device 20 uses the information of the ordered product identified in this way to perform the above comparison. Also, even if three or more image codes are identified within one rectangle, the ordered product can be identified by the same method as above.

<Other embodiments>
Various embodiments other than those described above can be adopted.
For example, the utilization rate of the order management system 1 can be displayed on the display unit 43 of the display device 40. The utilization rate may be the utilization rate in the store where the display device 40 is installed, or may be the utilization rate for each staff member of the store. The utilization rate of the order management system 1 can be indicated, for example, as the ratio of the number of uses of the order management system 1 to the number of issues (number of orders) of the above-mentioned customer numbers (order identification information) in a specified period (for example, one day or one month). The number of uses of the order management system 1 is indicated, for example, by the number of times that the management device 20 determines that preparation for providing the ordered products is complete.

It is also possible to measure the preparation time, which is the time from when a product order is received from a customer to when the product is ready to be served, and display this on the display unit 43 of the display device 40. In this case, the timing at which the product order is received may be, for example, the timing at which information on the customer's order is input via the input unit 14 of the terminal device 10. The display device 40 can calculate and display the preparation time for each order, the average preparation time for each staff member, the average preparation time for each store, or the average preparation time for each time period (for example, between 8:00 and 18:00).

Furthermore, when the display on the display device 40 does not change (the display freezes), it is also possible to display a video on the display unit 43 of the display device 40 to identify the cause. The video is changed by the control of the display device 40 (for example, an icon image of an animal moves). For example, if the display of the preparation status for product provision (for example, the display of an image as shown in FIG. 6) does not change even though the video display moves on the display unit 43 of the display device 40, the cause of the display not changing may be the communication network between the display device 40 and the management device 20. Also, if the video display does not move, the cause may be the display device 40.

The above-described embodiments are intended to facilitate understanding of the present invention, and are not intended to limit the present invention. The elements of the embodiments, as well as their arrangement, materials, conditions, shapes, sizes, etc., are not limited to those exemplified, and may be modified as appropriate. In addition, configurations shown in different embodiments may be partially substituted or combined.

1 Order management system, 10 Terminal device, 20 Management device, 30 Imaging device, 40 Display device, 50 Printer

Claims (17)

A display device that supports the provision of products,
displaying a list of products ordered by the customer based on order identification information included in an image code captured in the imaging area;
Based on the result of the imaging, displaying products included in the product list that are not provided in the imaging area in a manner different from that of the products provided in the imaging area;
displaying a message indicating that the products included in the product list have been provided to the imaging area; and
A control unit that executes
The control unit is a display device that displays the result of adding the second product provided in the imaging area as if the first product provided in the imaging area has been identified, even if a first product provided in the imaging area and displayed on the display device is no longer identified in the imaging area because a second product provided in the imaging area is overlapped on it . A display device that supports the provision of products,
displaying a list of products ordered by the customer based on order identification information included in an image code captured in the imaging area;
Based on the result of the imaging, displaying products included in the product list that are not provided in the imaging area in a manner different from that of the products provided in the imaging area;
displaying a message indicating that the products included in the product list have been provided to the imaging area; and
A control unit that executes
The control unit is a display device that displays the result of adding the number of products provided later to the imaging area to the number of products provided later to the imaging area when multiple orders for the same product are placed for the same product included in the product list and a product provided earlier to the imaging area and displayed on the display device among the multiple orders is no longer identified in the imaging area due to a product provided later to the imaging area among the multiple orders . A display device that supports the provision of products,
displaying a list of products ordered by the customer based on order identification information included in an image code captured in the imaging area;
Based on the result of the imaging, displaying products included in the product list that are not provided in the imaging area in a manner different from that of the products provided in the imaging area;
displaying a message indicating that the products included in the product list have been provided to the imaging area; and
A control unit that executes
A display device in which the image code is a code obtained by encoding information consisting of the order identification information and product identification information assigned according to the number of products included in the order identified by the order identification information . The display device according to claim 1 , wherein the product provided in the imaging area is identified based on product information included in the image code captured in the imaging area. The display device according to claim 1 , wherein the product provided in the imaging area is identified based on an image recognition process of the product on an image obtained by imaging the imaging area. The display device according to claim 1 , wherein, when the product provided in the imaging area is not included in the product list, the control unit executes a process of displaying a message indicating that the product provided in the imaging area is not included in the order . The display device according to claim 1 , wherein the control unit executes displaying the order identification information. The display device according to any one of claims 1 to 7, wherein the control unit executes the steps of displaying a list of products ordered by a first customer in a first area of the display unit and displaying a list of products ordered by a second customer in a second area of the display unit. A control device capable of communicating with a display device,
a storage unit that stores information on an ordered product identified based on order identification information included in an image code captured in an imaging area, and information on a product provided in the imaging area identified based on a result of the imaging;
A control unit.
The control unit is
displaying a list of the ordered items on the display device based on the information stored in the storage unit;
Based on the result of the imaging, displaying on the display device, among the products included in the list of ordered products, products that have not been provided in the imaging area in a manner different from that of the products that have been provided in the imaging area;
Based on the result of the imaging, displaying on the display device, among the products included in the list of ordered products, products that are no longer identified in the imaging area due to other products being overlapped therewith , in the same form as the products provided in the imaging area;
a control device that controls the display device to display a message indicating that the products included in the list of ordered products have been provided to the imaging area; The control device according to claim 9, wherein the control unit controls the display device to change the number of products provided in the imaging area when a new product is identified in the imaging area based on the imaging result . The control unit is
when none of the ordered items are identified in the imaging area based on the imaging result, removing the list of the ordered items from the display of the display device;
The control device according to claim 9 or 10, further comprising control means for controlling, when the ordered product that is no longer identified in the imaging area is identified again, to display a list of the ordered products on the display device based on the information stored in the memory unit . The control unit is
Identifying an object that fits within a predetermined shape in the imaging area;
comparing the ordered product identified based on the order identification information included in the image code within the predetermined shape with a product identified based on an image recognition result for the object within the predetermined shape;
and determining whether the product provided in the imaging area is correct based on a result of the comparison.
A control device according to any one of claims 9 to 11 . The control device according to claim 12 , wherein when a plurality of the image codes are identified within the predetermined shape, the control unit identifies the ordered product based on the order identification information included in the image code that is closest to a center of the predetermined shape. A display method executed by a control unit in a display device that supports the provision of products,
displaying a list of products ordered by the customer based on order identification information included in an image code captured in the imaging area;
Based on the result of the imaging, displaying products included in the product list that are not provided in the imaging area in a manner different from that of the products provided in the imaging area;
Even if a first product provided in the imaging area and displayed on the display device is no longer identified in the imaging area because a second product provided in the imaging area is overlapped thereon, a result of adding the second product provided in the imaging area to the first product provided in the imaging area is displayed as if the first product provided in the imaging area has been identified;
displaying a message indicating that the products included in the product list have been provided to the imaging area; and
Display methods including. The display device includes:
displaying a list of products ordered by the customer based on order identification information included in an image code captured in the imaging area;
Based on the result of the imaging, displaying products included in the product list that are not provided in the imaging area in a manner different from that of the products provided in the imaging area;
Even if a first product provided in the imaging area and displayed on the display device is no longer identified in the imaging area because a second product provided in the imaging area is overlapped thereon, a result of adding the second product provided in the imaging area to the first product provided in the imaging area is displayed as if the first product provided in the imaging area has been identified;
displaying a message indicating that the products included in the product list have been provided to the imaging area; and
A program for executing the above. A control method in a control device capable of communicating with a display device, comprising:
storing, in a storage unit, information on the ordered product identified based on order identification information included in an image code captured in the imaging area, and information on the product provided in the imaging area identified based on a result of the imaging;
The control unit:
displaying a list of the ordered items on the display device based on the information stored in the storage unit;
Based on the result of the imaging, displaying on the display device, among the products included in the list of ordered products, products that have not been provided in the imaging area in a manner different from that of the products that have been provided in the imaging area;
Based on the result of the imaging, displaying on the display device, among the products included in the list of ordered products, products that are no longer identified in the imaging area due to other products being overlapped therewith , in the same form as the products provided in the imaging area;
controlling the display device to display a message indicating that the products included in the list of ordered products have been provided to the imaging area;
A control method comprising: A control device capable of communicating with the display device,
storing, in a storage unit, information on the ordered product identified based on order identification information included in an image code captured in the imaging area, and information on the product provided in the imaging area identified based on a result of the imaging;
The control unit:
displaying a list of the ordered items on the display device based on the information stored in the storage unit;
Based on the result of the imaging, displaying on the display device, among the products included in the list of ordered products, products that have not been provided in the imaging area in a manner different from that of the products that have been provided in the imaging area;
Based on the result of the imaging, displaying on the display device, among the products included in the list of ordered products, products that are no longer identified in the imaging area due to other products being overlapped therewith , in the same form as the products provided in the imaging area;
controlling the display device to display a message indicating that the products included in the list of ordered products have been provided to the imaging area;
A program for executing.