JP7641182B2 - Information processing system, information processing method, and information processing program - Google Patents

Description

The present invention relates to an information processing system, an information processing method, and an information processing program for registering user attribute information.

Restaurants, retail stores, and other stores use images of users (customers) captured by cameras inside the facility to acquire attributes such as the age and gender of users for marketing purposes. For example, a system has been proposed in which cameras placed at the store entrances and aisles continuously capture images of customers moving around the store, and the customer's age, gender, and other attributes are determined from the captured images (see, for example, Patent Document 1).

Patent No. 4778532

Here, in store marketing, it is desirable to understand the relationship between a user's attributes and the products they have purchased. However, with conventional technology, it is difficult to determine the relationship between a user's attributes and the products they have purchased. Furthermore, a method in which a store clerk registers a user's attributes in association with the products they have purchased places a heavy burden on the store clerk, and there is a problem in that the accuracy is low because the determination is based on the store clerk's subjective judgment.

The object of the present invention is to provide an information processing system, information processing method, and information processing program that can easily register the attributes of users who use a store and the product information purchased by those users.

The information processing system according to the present invention is a system including a product acquisition processing unit that acquires purchased product information corresponding to information on purchased products of a user, an image acquisition processing unit that acquires an image of the user, an extraction processing unit that extracts a user image of the user from the image acquired by the image acquisition processing unit, a determination processing unit that determines attributes of the user based on the user image extracted by the extraction processing unit, and a registration processing unit that associates and registers attribute information indicating the attributes of the user determined by the determination processing unit with the purchased product information acquired by the product acquisition processing unit.

The information processing method according to the present invention is a method in which one or more processors execute a product acquisition step of acquiring purchased product information corresponding to information of a product purchased by a user, an image acquisition step of acquiring an image of the user, an extraction step of extracting a user image of the user from the image acquired in the image acquisition step, a determination step of determining attributes of the user based on the user image extracted in the extraction step, and a registration step of associating and registering attribute information indicating the user's attributes determined in the determination step with the purchased product information acquired in the product acquisition step.

The information processing program according to the present invention is a program for causing one or more processors to execute the following steps: a product acquisition step for acquiring purchased product information corresponding to information on purchased products of a user; an image acquisition step for acquiring an image of the user; an extraction step for extracting a user image of the user from the image acquired in the image acquisition step; a determination step for determining attributes of the user based on the user image extracted in the extraction step; and a registration step for registering attribute information indicating the attributes of the user determined in the determination step in association with the purchased product information acquired in the product acquisition step.

The present invention provides an information processing system, an information processing method, and an information processing program that can easily register the attributes of users who use a store and the product information of the users.

FIG. 1 is a block diagram showing the configuration of a transaction system according to an embodiment of the present invention. FIG. 2 is a diagram showing an example of an image analysis information table used in the accounting system according to an embodiment of the present invention. FIG. 3 is a diagram showing an example of an attribute information table used in the accounting system according to an embodiment of the present invention. FIG. 4 is a diagram showing an example of transaction information used in the transaction system according to an embodiment of the present invention. FIG. 5 is a diagram showing an example of purchased product information used in the accounting system according to the embodiment of the present invention. FIG. 6 is a diagram showing an example of user information used in the accounting system according to an embodiment of the present invention. FIG. 7 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. FIG. 8 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. FIG. 9 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. FIG. 10 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. FIG. 11 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. FIG. 12 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. FIG. 13 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. FIG. 14 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. FIG. 15 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. FIG. 16 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. FIG. 17 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. FIG. 18 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. FIG. 19 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. FIG. 20 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. FIG. 21 is a diagram showing an example of an accounting screen displayed on an accounting terminal according to an embodiment of the present invention. FIG. 22 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention. FIG. 23 is a flowchart showing an example of the procedure of a transaction process executed at a transaction terminal according to an embodiment of the present invention. FIG. 24 is a flowchart showing an example of the procedure of a transaction process executed at a transaction terminal according to an embodiment of the present invention. FIG. 25 is a diagram showing an example of an accounting screen displayed on the accounting terminal according to an embodiment of the present invention.

The following describes an embodiment of the present invention with reference to the attached drawings to facilitate understanding of the present invention. Note that the following embodiment is an example of the present invention and does not limit the technical scope of the present invention.

[Accounting system 100]
1, an accounting system 100 according to an embodiment of the present invention includes an accounting terminal 1, a camera 2, and a database DB. The accounting terminal 1, the camera 2, and the database DB are capable of communicating with each other via a communication network N1, such as a LAN, the Internet, a WAN, or a public telephone line.

Camera 2 is a digital camera that captures an image of a subject and outputs it as digital image data. Camera 2 has a communication function that transmits the digital image data to accounting terminal 1 through communication network N1. There may be one camera 2 or multiple cameras 2. Specifically, camera 2 is installed in a store and captures a predetermined area including store users (customers). For example, camera 2 is installed at a store's cash register and captures a predetermined area inside the store from the cash register. Camera 2 captures the predetermined area at a predetermined frame rate and transmits the image data to accounting terminal 1. The predetermined area is, for example, an area in which a user making a payment can be captured. The predetermined area may also be, for example, the entire area of the store. In the present invention, it is also possible to use a camera already installed in the store as camera 2. Camera 2 may also be mounted on accounting terminal 1.

The accounting terminal 1 is a device (e.g., a POS terminal) that is installed in a store and that performs accounting for the items purchased by a user. The accounting terminal 1 may perform accounting by accepting operations from a store clerk or from the user. The accounting terminal 1 also determines the attributes (age, gender, etc.) of the user performing the accounting based on an image of a specified area captured by the camera 2, and registers attribute information indicating the user's attributes in association with purchased item information regarding the items purchased by the user. Examples of stores include restaurants and retail stores. In the following embodiment, a restaurant will be described as an example of a store.

In this embodiment, the accounting terminal 1 alone corresponds to the information processing system according to the present invention, but the information processing system according to the present invention may be configured to include the accounting terminal 1, the camera 2, and the database DB. Also, the accounting terminal 1, the camera 2, and the database DB may be integrated into one device.

[Accounting terminal 1]
As shown in Figure 1, the accounting terminal 1 is an information processing device (e.g., a POS terminal) equipped with a control unit 11, a memory unit 12, an operation display unit 13, and a communication unit 14. Note that the accounting terminal 1 is not limited to a single computer, but may be a computer system in which multiple computers work together. Furthermore, the various processes executed by the accounting terminal 1 may be executed in a distributed manner by one or multiple processors. The accounting terminal 1 is an example of an information processing system of the present invention.

The communication unit 14 is a communication interface that connects the accounting terminal 1 to the communication network N1 via a wired or wireless connection and performs data communication in accordance with a specified communication protocol with one or more external devices such as the camera 2 via the communication network N1.

The operation display unit 13 is a user interface that includes a display unit such as a liquid crystal display or an organic EL display that displays various information, and an operation unit such as a mouse, keyboard, or touch panel that accepts operations.

The storage unit 12 is a non-volatile storage unit including a semiconductor memory, a hard disk drive (HDD), or a solid state drive (SSD) that stores various information. Specifically, the storage unit 12 stores an image analysis information table D1 that registers the analysis results of the captured image captured by the camera 2, an attribute information table D2 that registers the attributes of users (customers) extracted from the captured image, and the like. FIG. 2 is a diagram showing an example of the image analysis information table D1, and FIG. 3 is a diagram showing an example of the attribute information table D2. The captured image includes an image of one or more users making a payment at the cash register.

2, in the image analysis information table D1, information such as a user number, face area (horizontal), face area (vertical), gender, age, and display flag is registered for each face number set in the face image of the user extracted from the captured image. The user number is user identification information set in response to an operation by a store clerk to select a user image included in the captured image. The face area (horizontal and vertical) is information indicating the position and range of the face image of the user included in the captured image. The gender is the gender of the user determined by the control unit 11 based on the user image included in the captured image. The age is the age of the user determined by the control unit 11 based on the user image included in the captured image. The display flag is information indicating whether the user's attributes have been displayed on the accounting screen (see FIG. 7, etc.) displayed on the operation display unit 13 of the accounting terminal 1. When the control unit 11 acquires the captured image, it analyzes the captured image (image analysis) and registers each piece of information in the image analysis information table D1.

As shown in FIG. 3, in the attribute information table D2, information such as gender, age, generation, and revision flag is registered for each user number. Each time a store clerk selects a user image in the captured image (see FIG. 9, etc.), the control unit 11 registers the gender, age, and generation in association with the user number corresponding to the order of the selection operation. The revision flag is information indicating whether or not an operation (the revision operation of the present invention) (see FIG. 13, etc.) is being performed (currently being revised) to correct at least one of the registered gender and age.

The control unit 11 registers each piece of information in the image analysis information table D1 and the attribute information table D2 based on the captured image.

In another embodiment, one or both of the image analysis information table D1 and the attribute information table D2 may be stored in another server accessible from the accounting terminal 1 via the communication network N1. In this case, the control unit 11 of the accounting terminal 1 may obtain various information from the server as necessary and execute various processes such as the accounting process described below (see Figures 23 and 24).

The control unit 11 also executes accounting processing based on various information stored in the database DB. The database DB stores data such as accounting information D3, purchased product information D4, and user information D5. Figure 4 is a diagram showing an example of accounting information D3, Figure 5 is a diagram showing an example of purchased product information D4, and Figure 6 is a diagram showing an example of user information D5.

As shown in FIG. 4, transaction information D3 includes information such as the transaction number, number of items purchased, and number of people in a group for each transaction. The transaction number is identification information assigned to each transaction. The number of items purchased is the total number of items included in the transaction. The number of people in a group is the total number of people such as families and group customers, for example the number of people occupying a table in a restaurant.

As shown in FIG. 5, purchased item information D4 includes information such as the transaction number and purchased items (ordered items) for each transaction. In the case of a group that includes multiple users, multiple purchased items are associated with and registered for the same transaction number, for example.

The accounting information D3 and purchased product information D4 may be stored in the database DB based on information entered into an order terminal (not shown), for example.

As shown in FIG. 6, user information D5 includes information such as gender, age, and generation for each transaction. The control unit 11 registers the user attributes registered in attribute information table D2 in association with the corresponding transaction number in user information D5.

The storage unit 12 also stores control programs such as an accounting program (corresponding to the information processing program of the present invention) for causing the control unit 11 to execute the accounting process described below (see Figures 23 and 24). For example, the accounting program is non-temporarily recorded on a computer-readable recording medium such as a CD or DVD, and is read by a reading device (not shown) such as a CD drive or DVD drive provided in the accounting terminal 1 and stored in the storage unit 12.

The control unit 11 has control devices such as a CPU, a ROM, and a RAM. The CPU is a processor that executes various arithmetic processes. The ROM is a non-volatile storage unit in which control programs such as a BIOS and an OS for causing the CPU to execute various arithmetic processes are stored in advance. The RAM is a volatile or non-volatile storage unit that stores various information, and is used as a temporary storage memory (work area) for the various processes executed by the CPU. The control unit 11 controls the accounting terminal 1 by having the CPU execute various control programs that are stored in advance in the ROM or the storage unit 12.

Specifically, as shown in FIG. 1, the control unit 11 includes various processing units such as a product acquisition processing unit 111, an image acquisition processing unit 112, an extraction processing unit 113, a judgment processing unit 114, a registration processing unit 115, a display processing unit 116, and a reception processing unit 117. The control unit 11 functions as the various processing units by executing various processes according to the accounting program with the CPU. Some or all of the processing units included in the control unit 11 may be configured with electronic circuits. The accounting program may be a program for causing multiple processors to function as the various processing units.

The product acquisition processing unit 111 acquires purchased product information corresponding to the information of the purchased product of the user. Specifically, the product acquisition processing unit 111 references the database DB to acquire accounting information D3 and purchased product information D4 corresponding to the user. Accounting information D3 (see FIG. 5) and purchased product information D4 (see FIG. 6) are examples of the purchased product information. For example, at a restaurant, when a user finishes their meal and pays at the register, the user hands the receipt to the cashier, and the cashier reads the information code on the receipt using a reading device or the like. As a result, the product acquisition processing unit 111 acquires the purchased product information of the user based on the accounting number (see FIG. 5 and FIG. 6) corresponding to the information code. The accounting number is an example of purchased product information of the present invention. The product acquisition processing unit 111 is also an example of a product acquisition processing unit of the present invention.

The image acquisition processing unit 112 acquires an image of a predetermined area including a customer in front of the cash register of the store, captured by the camera 2. The image acquisition processing unit 112 is an example of the image acquisition processing unit of the present invention.

The extraction processing unit 113 extracts a user image of the user from the captured image acquired by the image acquisition processing unit 112. Specifically, the extraction processing unit 113 extracts a facial image of the user from the captured image. If the captured image includes multiple users, the extraction processing unit 113 extracts a facial image for each user. When the extraction processing unit 113 extracts a facial image, it associates the facial image with a face number and registers information on the facial area of the facial image in the image analysis information table D1 (see FIG. 2). The extraction processing unit 113 is an example of an extraction processing unit of the present invention.

The determination processing unit 114 determines the attributes of the user based on the user image extracted by the extraction processing unit 113. For example, the determination processing unit 114 estimates the gender and age of the user from the face image using a trained model generated by machine learning. If the captured image includes multiple users, the determination processing unit determines the gender and age of each user. After determining the attributes (gender and age) of the user, the determination processing unit 114 registers the gender and age information in the image analysis information table D1 (see FIG. 2) in association with the face number. The determination processing unit 114 is an example of a determination processing unit of the present invention.

The registration processing unit 115 associates and registers attribute information indicating the attributes of the user determined by the determination processing unit 114 with the purchased product information acquired by the product acquisition processing unit 111. For example, the registration processing unit 115 stores in the database DB user information D5 (see FIG. 6) in which the account number acquired by the product acquisition processing unit 111 based on the information code of the slip is associated with the user's gender and age information. The registration processing unit 115 also registers the generation corresponding to the age determined by the determination processing unit 114 in the user information D5. The registration processing unit 115 is an example of a registration processing unit of the present invention.

The display processing unit 116 displays the captured image acquired by the image acquisition processing unit 112 on the operation display unit 13, and also displays attribute information of the user on the operation display unit 13 in association with the user image of the user extracted from the captured image.

Figure 7 shows an example of an accounting screen displayed on the operation display unit 13. The accounting screen displays information on all items that can be ordered at the restaurant, an image P1 of the users (here, a group of three) making the payment, order information related to the items ordered (purchased products) of the group, and an attribute editing field P2 for displaying and editing the attributes of each user. As shown in Figure 8, the display processing unit 116 displays the image P1 on the accounting screen, and also displays the gender and age (attribute information) of each user determined from the image P1 near the user image (e.g., a facial image). This allows the store clerk to know the gender and age of the users on the accounting screen. The display processing unit 116 is an example of a display processing unit of the present invention.

The reception processing unit 117 accepts a selection operation to select the user image in the captured image P1 of the checkout screen displayed on the operation display unit 13. For example, as shown in FIG. 9, the store clerk touches the user image (face image) of one user (e.g., the user on the far right) included in the captured image P1. This causes the reception processing unit 117 to accept the store clerk's touch operation.

When the reception processing unit 117 receives the touch operation (see FIG. 9), the registration processing unit 115 registers "1" in the image analysis information table D1 as the user number associated with the gender and age of the user of the touched user image. The display processing unit 116 also displays the user number "1" near the face image of the captured image P1 on the checkout screen (see FIG. 10), and displays the gender and generation (age) in the field corresponding to the number "1" in the attribute editing field P2. In the example shown in FIG. 9 and FIG. 10, the display processing unit 116 displays "male" and "40-49 (41)" in association with the number "1" in the attribute editing field P2. The image of the user number "1" displayed on the checkout screen is an example of a sequence image of the present invention. Note that the sequence image of the present invention is not limited to numbers, and may be letters or symbols such as the alphabet.

When the display processing unit 116 displays the user's attributes in the attribute editing field P2, the registration processing unit 115 registers "ON (already displayed)" in the display flag corresponding to the user number "1" in the image analysis information table D1 (see FIG. 2). The registration processing unit 115 also registers the user's information (here, "male", "41", "40-49") in the gender, age, and generation corresponding to the user number "1" in the attribute information table D2 (see FIG. 3).

After the attributes of one user included in the captured image P1 are registered, the store clerk then touches the user image (face image) of another user (e.g., the user on the far left) included in the captured image P1, as shown in FIG. 11. The reception processing unit 117 accepts the touch operation by the store clerk.

When the reception processing unit 117 receives the touch operation (see FIG. 11), the registration processing unit 115 registers "2" in the image analysis information table D1 as the user number associated with the gender and age of the user of the touched user image. The display processing unit 116 also displays the user number "2" on the checkout screen near the face image of the captured image P1 (see FIG. 11), and displays the gender and generation (age) in the field corresponding to the number "2" in the attribute editing field P2. In the example shown in FIG. 11, the display processing unit 116 registers "female" and "30-39 (38)" in association with the number "2" in the attribute editing field P2.

When the display processing unit 116 displays the user's attributes in the attribute editing field P2, the registration processing unit 115 registers "ON (already displayed)" in the display flag corresponding to the user number "2" in the image analysis information table D1 (see FIG. 2). The registration processing unit 115 also registers the user's information (here, "female," "38," and "30-39") in the gender, age, and generation corresponding to the user number "2" in the attribute information table D2 (see FIG. 3).

Next, as shown in FIG. 12, the store clerk touches the user image (face image) of the remaining user (e.g., the user in the center) included in the captured image P1. The reception processing unit 117 accepts the touch operation by the store clerk.

When the reception processing unit 117 receives the touch operation (see FIG. 12), the registration processing unit 115 registers "3" in the image analysis information table D1 as the user number associated with the gender and age of the user of the touched user image. The display processing unit 116 also displays the user number "3" on the checkout screen near the face image of the captured image P1 (see FIG. 12), and displays the gender and generation (age) in the field corresponding to the number "3" in the attribute editing field P2. In the example shown in FIG. 12, the display processing unit 116 registers "female" and "10-19 (11)" in association with the number "3" in the attribute editing field P2.

When the display processing unit 116 displays the user's attributes in the attribute editing field P2, the registration processing unit 115 registers "ON (already displayed)" in the display flag corresponding to the user number "3" in the image analysis information table D1 (see FIG. 2). The registration processing unit 115 also registers the user's information (here, "female," "11," and "10-19") in the gender, age, and generation corresponding to the user number "3" in the attribute information table D2 (see FIG. 3).

In this way, the attributes of the user (gender and age) included in the captured image are associated with the purchased product information of the user and registered. In the above example, "Fried chicken set meal 700 yen", "Pork cutlet bowl 600 yen", and "Ramen 550 yen" are registered in association with user number "1" (male 40-49 (41)), user number "2" (female 30-39 (38)", and user number "3" (female 10-19 (11)). When the registration operation is completed, the store clerk touches the "OK" button on the accounting screen shown in FIG. 12. When the store clerk touches the "OK" button, the control unit 11 executes the accounting process for the purchased products.

Here, the store clerk can modify the gender and age of the user registered in the attribute editing field P2 on the checkout screen. The reception processing unit 117 accepts a modification operation to modify the attribute information on the checkout screen. Furthermore, the display processing unit 116 displays the modified attribute information when the reception processing unit 117 accepts the modification operation. For example, as shown in FIG. 13, when the store clerk wants to modify at least one of the gender and age of user number "1", the "Modify" button of user number "1" is touched. This causes the reception processing unit 117 to accept the modification operation. Furthermore, the registration processing unit 115 sets the modification flag of user number "1" in the attribute information table D2 (see FIG. 3) to "ON". Furthermore, the display processing unit 116 changes the color of the area of the touched "Modify" button (see FIG. 14). After that, when the store clerk touches "male" of user number "1" (see FIG. 14), the display processing unit 116 modifies "male" to "female" and displays it. In addition, the registration processing unit 115 corrects the "male" for user number "1" in the image analysis information table D1 and the attribute information table D2 to "female" and registers the user.

When the store clerk touches the "Edit" button for user number "1" and then touches "40-49 (41)" (see FIG. 15), the display processing unit 116 displays a numeric keypad for inputting (selecting) numbers, as shown in FIG. 16. When the store clerk inputs "50" using the numeric keypad and touches the "OK" button (see FIG. 17), the display processing unit 116 corrects "40-49 (41)" to "50-59 (50)" and displays it, as shown in FIG. 18. The registration processing unit 115 also corrects "40-49 (41)" for user number "1" in the image analysis information table D1 and the attribute information table D2 to "50-59 (50)" and registers it.

As described above, the registration processing unit 115 registers the user's attribute information in the image analysis information table D1 and the attribute information table D2. The registration processing unit 115 also registers the attribute information in the accounting information D3 (see FIG. 4). That is, the registration processing unit 115 registers the user's gender and age in association with the purchased product information. The display processing unit 116 also displays the user number information, the attribute information, and the purchased product information in association with each other on the accounting screen on which the captured image P1 is displayed.

In the above embodiment, the determination processing unit 114 determines the attributes (gender and age) of the user from the user image included in the captured image, and the registration processing unit 115 registers the user number in association with the attribute information corresponding to the user image selected by the user (store clerk). In another embodiment, the registration processing unit 115 may automatically register the user number in association with the attribute information corresponding to each user image included in the captured image, without relying on the selection operation of the store clerk.

In addition, in the other embodiment, when the determination processing unit 114 cannot determine the attribute of the user, the registration processing unit 115 may register the attribute based on the operation of the store clerk. For example, as shown in FIG. 19, when the determination processing unit 114 cannot determine (estimate) the gender and age of the user of the rightmost user image included in the captured image P1, the gender and age information is not displayed on the user image. In this case, when the store clerk touches the user image, the display processing unit 116 displays the user number "4" near the face image of the user image and changes the color of the "Modify" button area of the user number "4" (see FIG. 20). In addition, the display processing unit 116 displays the provisional gender ("male") and age ("00-09 (00)") in the attribute editing field P2 (see FIG. 20). In addition, when the store clerk touches the gender and age to input the gender and age, the display processing unit 116 corrects the gender and age. In addition, the registration processing unit 115 registers the gender and age to the user number "4" in the image analysis information table D1 and the attribute information table D2.

In addition, a user who does not belong to the group may appear in the captured image P1. For example, as shown in FIG. 21, the leftmost user image included in the captured image P1 is an image of a user who does not belong to the same group. In this case, the store clerk can perform an operation to delete the information of the user. Specifically, the store clerk touches the user number "2" of the user image at the left end of the captured image P1 on the checkout screen shown in FIG. 21. When the store clerk touches the user number "2", the display processing unit 116 deletes the user number "2", and the registration processing unit 115 deletes the information corresponding to the user number "2" in the image analysis information table D1 and the attribute information table D2. In addition, as shown in FIG. 22, the display processing unit 116 raises the numbers "3" and "4" of the other user images in the captured image P1 to the numbers "2" and "3", respectively, and raises the attribute information of the numbers "3" and "4" in the attribute editing field P2 to the numbers "2" and "3". Additionally, the registration processing unit 115 increments the attribute information of user numbers "3" and "4" in the image analysis information table D1 and attribute information table D2 to user numbers "2" and "3."

In this way, the reception processing unit 117 accepts an operation to delete the user number displayed in association with the user image on the accounting screen. The display processing unit 116 then deletes the user number information and the attribute information that were displayed in association with each other in the attribute editing field P2 on the accounting screen.

[Accounting]
The accounting process executed by the control unit 11 of the accounting terminal 1 will be described below. For example, the control unit 11 starts the accounting process by starting execution of the accounting program by acquiring an image of a predetermined area in front of a cash register captured by the camera 2. Note that when one or more cameras 2 capture images of the respective predetermined areas of multiple cash registers, the control unit 11 may execute the accounting process individually in parallel for each acquired image.

The present invention can be understood as an invention of an accounting method that executes one or more steps included in the accounting process. The accounting method is an example of the information processing method of the present invention.

Note that one or more steps included in the accounting process described here may be omitted as appropriate. Furthermore, the steps in the accounting process may be executed in a different order as long as the same action and effect is achieved. Furthermore, although an example is described here in which the control unit 11 executes each step in the accounting process, in other embodiments, multiple processors may execute each step in the accounting process in a distributed manner.

Figure 23 is a flowchart showing an example of the procedure for attribute determination processing included in the transaction processing. During the transaction processing, the control unit 11 first executes attribute determination processing to determine the attributes of the user based on the captured image.

In step S1, the control unit 11 acquires purchased product information corresponding to the information of the product purchased by the user. Specifically, the control unit 11 references the database DB to acquire accounting information D3 (see FIG. 4) and purchased product information D4 (see FIG. 5) corresponding to the user. For example, the control unit 11 acquires an accounting number from a slip, and acquires accounting information D3 and purchased product information D4 associated with the accounting number. Step S1 is an example of a product acquisition step of the present invention.

Next, in step S2, the control unit 11 acquires an image P1 of a predetermined area of the store, including a customer standing in front of the cash register, captured by the camera 2. The control unit 11 displays the acquired image P1 on the checkout screen (see FIG. 7). Step S2 is an example of an image acquisition step of the present invention.

Next, in step S3, the control unit 11 sets the face number in the image analysis information table D1 (see FIG. 2) to "0" and clears (initializes) the gender, age, and display flag.

Next, in step S4, the control unit 11 starts scanning (scanning process) the acquired captured image P1. For example, the control unit 11 scans the captured image P1 from the right side in the horizontal direction.

Next, in step S5, the control unit 11 determines whether or not a face image has been detected from the captured image P1. If the control unit 11 detects the face image from the captured image P1 (S5: Yes), the process proceeds to step S6. On the other hand, if the control unit 11 does not detect the face image from the captured image P1 (S5: No), the process proceeds to step S10.

In step S6, the control unit 11 extracts the face image from the captured image P1 and registers the face number "1" and the face area (horizontal and vertical directions) in the image analysis information table D1 (see FIG. 2). Steps S5 and S6 are an example of an extraction step of the present invention.

Next, in step S7, the control unit 11 determines (estimates) the gender and age based on the extracted face image. Step S7 is an example of the determination step of the present invention.

Next, in step S8, the control unit 11 registers the determined gender and age in association with the face number "1" in the image analysis information table D1. Here, the control unit 11 registers "male" and "41" in association with the face number "1."

Next, in step S9, the control unit 11 displays the determined gender and age in the captured image P1 on the checkout screen (see FIG. 8). Note that for convenience, the genders and ages of three users are displayed in FIG. 8, but in step S9, only the rightmost user, "Male 41," is displayed.

Next, in step S10, the control unit 11 determines whether scanning has been completed up to the end (here, the left end) of the captured image P1. When the control unit 11 completes scanning (S10: Yes), it ends the attribute determination process. On the other hand, when the control unit 11 has not completed scanning (S10: No), it proceeds to step S101 and continues scanning.

When the control unit 11 continues scanning the captured image P1 and detects the next facial image (S5: Yes), it executes the above-mentioned processing of steps S6 to S9. In this way, the control unit 11 repeats the above-mentioned processing until it detects no more facial images from the captured image P1. When the control unit 11 ends the attribute determination processing, it executes the attribute registration processing described below. Figure 8 shows the state where the control unit 11 has finished the attribute registration processing for the captured image P1.

Figure 24 is a flowchart showing an example of the procedure for the transaction process. When the attribute determination process (see Figure 23) is completed, the control unit 11 executes an attribute registration process for registering the attributes of the user in response to the operation of the user (store clerk), an attribute editing process for modifying the registered attributes, and a transaction process.

In step S11, the control unit 11 sets the user number to "0" in the image analysis information table D1 and the attribute information table D2, and sets the displayed flag and the correction flag to "OFF."

Next, in step S12, the control unit 11 determines whether or not a touch input by a store clerk has been detected on the checkout screen (see FIG. 8). If the control unit 11 detects the touch input (S12: Yes), the process proceeds to step S13. The control unit 11 waits until the touch input is detected (S12: No).

In step S13, the control unit 11 determines whether the position of the touch input is a face image in the captured image P1. If the position of the touch input is the face image (S13: Yes), the process proceeds to step S14. If the position of the touch input is not the face image (S13: No), the process proceeds to step S21.

In step S14, the control unit 11 determines whether the attributes (gender and age) registered in the image analysis information table D1 in association with the face number corresponding to the face image have already been displayed in the attribute editing field P2 (see FIG. 9) of the checkout screen. If the attributes have not been displayed in the attribute editing field P2 (S14: No), the process proceeds to step S15. On the other hand, if the attributes have been displayed in the attribute editing field P2 (S14: Yes), the process returns to step S12.

In step S15, the control unit 11 registers "1" in the user number corresponding to the attribute in the image analysis information table D1 (see FIG. 2).

Next, in step S16, the control unit 11 displays the user number "1" near the face image (see Figure 10).

Next, in step S17, the control unit 11 displays the gender, age, and generation in the column corresponding to the user number "1" in the attribute editing column P2 (see FIG. 9) (see FIG. 10).

Next, in step S18, the control unit 11 sets the displayed flag in the image analysis information table D1 to "ON."

Next, in step S19, the control unit 11 registers the gender, age, and generation in association with the user number "1" in the attribute information table D2 (see FIG. 3). The control unit 11 then returns to step S12 and repeats the above-mentioned process. The store clerk touches all the user images included in the captured image P1 to display the attribute information of each user in the attribute editing field P2.

If the position of the touch input is not the face image in step S13 (S13: No), the process proceeds to step S21, where the control unit 11 determines whether the position of the touch input is the position of the correction button in the attribute editing field P2 (see FIG. 13). If the position of the touch input is the position of the correction button (S21: Yes), the process proceeds to step S22. If the position of the touch input is not the position of the correction button (S21: No), the process proceeds to step S31.

In step S22, the control unit 11 accepts an operation from the store clerk to modify an attribute in the attribute editing field P2, and executes a modification process to modify the attribute.

For example, as shown in FIG. 13, when a store clerk touches the correction button for user number "1," the control unit 11 sets the correction flag for user number "1" in attribute information table D2 to "ON," and changes the color of the area of the touched correction button on the checkout screen (see FIG. 14). The control unit 11 may also light up or blink the correction button.

When the correction flag is ON and the store clerk touches the "male" of user number "1" in the attribute editing field P2 (see FIG. 14), the control unit 11 corrects "male" to "female" and displays it. The control unit 11 also corrects "male" of user number "1" in the image analysis information table D1 and the attribute information table D2 to "female" and registers it.

When the correction flag is ON and the store clerk touches "40-49 (41)" for user number "1" in the attribute editing field P2 (see FIG. 15), the control unit 11 displays a numeric keypad for inputting (selecting) numbers, as shown in FIG. 16. When the store clerk inputs "50" using the numeric keypad and touches the OK button (see FIG. 17), the control unit 11 corrects "40-49 (41)" to "50-59 (50)" and displays it, as shown in FIG. 18. The control unit 11 also corrects "40-49 (41)" for user number "1" in the image analysis information table D1 and the attribute information table D2 to "50-59 (50)" and registers it.

When the clerk has completed editing the attribute editing field P2, the control unit 11 sets the correction flag to OFF and returns the color of the correction button to its original color.

If the position of the touch input is not the position of the correction button in step S13 (S21: No), the process proceeds to step S31, where the control unit 11 determines whether the position of the touch input is the position of the OK button (see FIG. 12). If the position of the touch input is the position of the OK button (S31: Yes), the process proceeds to step S32. If the position of the touch input is not the position of the OK button (S31: No), the process ends.

In step S32, the control unit 11 registers the attribute information in the user information D5 (see FIG. 6). For example, the control unit 11 registers the gender and age information of the user in association with the accounting number obtained based on the information code of the user's slip in the user information D5. Step S32 is an example of a registration step of the present invention.

In the final step S33, the control unit 11 executes the accounting process for the purchased items. The control unit 11 executes the accounting process for the purchased items by the user in the manner described above.

In the above-described checkout process, the control unit 11 determines the user's attributes (gender and age) from the user image included in the captured image P1, and registers a user number in association with the attribute information corresponding to the user image selected by the store clerk. In another embodiment, the control unit 11 may automatically register a user number in association with the attribute information corresponding to each user image included in the captured image P1, without relying on the selection operation of the store clerk.

In addition, in the other embodiment, the control unit 11 may register attributes based on the operation of the store clerk when the control unit 11 cannot determine the attributes of the user. For example, as shown in FIG. 19, when the control unit 11 cannot determine the gender and age of the user of the rightmost user image included in the captured image, the gender and age information is not displayed on the user image. In this case, when the store clerk touches the user image, the control unit 11 displays the user number "4" near the face image of the user image and changes the color of the correction button area for the user number "4" (see FIG. 20). When the store clerk touches the gender and age to input the gender and age, the control unit 11 corrects the gender and age. The control unit 11 also registers the gender and age to the user number "4" in the image analysis information table D1 and the attribute information table D2.

21, when the store clerk touches the user number "2" of the user image on the left side of the captured image P1, the control unit 11 deletes the user number "2" of the user image and deletes the information corresponding to the user number "2" in the image analysis information table D1 and the attribute information table D2. As shown in FIG. 22, the control unit 11 also increments the numbers "3" and "4" of other user images in the captured image P1 to "2" and "3", respectively, and increments the attribute information of the numbers "3" and "4" in the attribute editing field P2 to "2" and "3". The control unit 11 also increments the attribute information of the user numbers "3" and "4" in the image analysis information table D1 and the attribute information table D2 to "2" and "3".

As described above, the accounting system 100 according to this embodiment acquires purchased product information including information on products purchased by a user, acquires an image P1 of the user, and extracts a user image of the user from the image P1. The accounting system 100 also determines the attributes (gender and age) of the user based on the extracted user image, and associates and registers the acquired purchased product information with attribute information indicating the determined attributes of the user. This makes it possible to easily register the attributes of users who use a store and the purchased product information of the users. In addition, it is not necessary for a store clerk to register the user's attribute information. This reduces the workload of the store clerk and allows accurate attribute information to be registered. This improves the accuracy of the store's market research and marketing.

The accounting system 100 of the present invention is not limited to the above-described embodiment. The accounting system 100 may have the following forms:

As another embodiment, for example as shown in FIG. 25, when the control unit 11 is unable to determine attributes such as the gender or age of a user from the user image (face image) included in the captured image P1, it may display a "?" near the face image of the user image. In this case, for example, a store clerk touches the face image with a "?" on the checkout screen shown in FIG. 25 to input "age." This allows the control unit 11 to register and display the age corresponding to the user.

1: accounting terminal 2: camera 100: accounting system 111: product acquisition processing unit 112: image acquisition processing unit 113: extraction processing unit 114: judgment processing unit 115: registration processing unit 116: display processing unit 117: reception processing unit D1: image analysis information table D2: attribute information table D3: accounting information D4: purchased product information D5: user information P1: captured image P2: attribute editing field

Claims (9)

a product acquisition processing unit that acquires purchased product information corresponding to information on a product purchased by a user;
an image acquisition processing unit that acquires an image of the user;
an extraction processing unit that extracts a user image of the user from the captured image acquired by the image acquisition processing unit;
a determination processing unit that determines an attribute of the user based on the user image extracted by the extraction processing unit;
a registration processing unit that associates attribute information indicating the attributes of the user determined by the determination processing unit with the purchased product information acquired by the product acquisition processing unit and registers the associated attribute information;
a display processing unit that displays the captured image acquired by the image acquisition processing unit on a display unit, and that displays the attribute information of the user on the display unit in association with the user image of the user extracted from the captured image;
a reception processing unit that receives a selection operation for selecting the user image from the captured image displayed on the display unit;
Equipped with
The display processing unit displays an order image indicating an order of the selection operation in association with the selected user image . The display processing unit displays the attribute information in the vicinity of the user image.
The information processing system according to claim 1 . The registration processing unit registers the attribute information and the purchased product information in association with each other.
3. The information processing system according to claim 1 or 2 . the display processing unit causes the display unit on which the captured image is displayed to display the order information, the attribute information, and the purchased product information in association with each other.
The information processing system according to claim 3 . the reception processing unit receives a correction operation for correcting the attribute information on the display unit,
the display processing unit displays the corrected attribute information when the reception processing unit receives the correction operation.
5. The information processing system according to claim 4 . the reception processing unit receives an operation to delete the sequence image displayed in association with the user image on the display unit,
the display processing unit deletes the order information and the attribute information that are displayed in association with each other on the display unit.
5. The information processing system according to claim 4 . The extraction processing unit extracts user images of each of a plurality of users from the captured image,
The determination processing unit determines an attribute of the user for each of the user images.
The information processing system according to any one of claims 1 to 6 . One or more processors
A product acquisition step of acquiring purchased product information corresponding to information on a product purchased by a user;
an image acquisition step of acquiring a captured image of the user;
an extraction step of extracting a user image of the user from the captured image acquired in the image acquisition step;
a determination step of determining attributes of the user based on the user image extracted by the extraction step;
a registration step of registering attribute information indicating the attributes of the user determined in the determination step and the purchased product information acquired in the product acquisition step in association with each other;
a display step of displaying the captured image acquired in the image acquisition step on a display unit, and displaying the attribute information of the user on the display unit in association with the user image of the user extracted from the captured image;
a receiving step of receiving a selection operation for selecting the user image from the captured image displayed on the display unit;
Run
In the display step, an order image indicating an order of the selection operation is displayed in association with the selected user image . A product acquisition step of acquiring purchased product information corresponding to information on a product purchased by a user;
an image acquisition step of acquiring a captured image of the user;
an extraction step of extracting a user image of the user from the captured image acquired in the image acquisition step;
a determination step of determining attributes of the user based on the user image extracted by the extraction step;
a registration step of registering attribute information indicating the attributes of the user determined in the determination step and the purchased product information acquired in the product acquisition step in association with each other;
a display step of displaying the captured image acquired in the image acquisition step on a display unit, and displaying the attribute information of the user on the display unit in association with the user image of the user extracted from the captured image;
a receiving step of receiving a selection operation for selecting the user image from the captured image displayed on the display unit;
on one or more processors ,
An information processing program that displays, in the display step, an order image indicating an order of the selection operation in association with the selected user image .