JP7346869B2 - Image processing program, image processing device, and image processing method - Google Patents

Description

The present invention relates to an image processing program, an image processing device, and an image processing method.

BACKGROUND ART Estimation techniques have been known for detecting a customer's line of sight in a store and estimating products that the customer is interested in from among displayed products. According to the estimation technology, a salesperson or the like can recommend products that meet the customer's needs when serving customers.

On the other hand, in the sale of beauty products such as cosmetics, for example, beauty staff and the like serve customers while actually applying make-up to customers in a dedicated booth or the like. In such a sales scene, it is important for beauty staff and the like to determine which part of the customer's face reflected on the dressing table is of interest to the seated customer.

Japanese Patent Application Publication No. 2018-206354

However, the position of the customer's face reflected on the dressing table is not constant and changes as the customer changes his or her posture. Therefore, by simply detecting the customer's line of sight, it is not possible to determine which part of the customer's face is actually looking at the mirror, and it is not possible to estimate which parts of the face the customer is interested in. Can not.

In one aspect, the purpose is to determine which part of the face the user is looking at.

According to one aspect, the image processing program includes:
Obtaining the position of the user's line of sight directed toward a mirror image of the user or a display image of the user captured by an imaging device;
a position of the user's face in the mirror image, or a part of the user's face in the mirror image that corresponds to the position of the line of sight obtained by identifying the position of the user's face in the display image; , or determining a part of the user's face in the displayed image;
estimating the current object of interest of the user by determining a temporal change in the number of times the user looks at each part of the face based on the determined result;
Have a computer perform a process.

The part of the face that the user is looking at can be determined.

FIG. 1 is a first diagram illustrating an example of a system configuration of a target of interest estimation system. FIG. 3 is a diagram for explaining the photographing range of the imaging device and the position coordinates of the customer's face within the image data and within the dressing table. FIG. 2 is a diagram for explaining the measurement range of a line-of-sight sensor and the position coordinates of a customer's line of sight within a dressing table. FIG. 3 is a diagram for explaining the relationship between the positional coordinates of a customer's face and the positional coordinates of the customer's line of sight within the dressing table. 1 is a diagram illustrating an example of a hardware configuration of an image processing device. FIG. 1 is a first diagram showing an example of a functional configuration of an image processing device. It is a figure showing an example of data accumulated in a data storage part. FIG. 3 is a diagram for explaining facial parts. It is a 1st flowchart which shows the flow of an interest object estimation process. 12 is a second flowchart showing the flow of interest object estimation processing. It is a figure showing an example of a total result. FIG. 2 is a second diagram showing an example of the system configuration of the object-of-interest estimation system. FIG. 3 is a diagram for explaining the photographing range of the imaging device and the position coordinates of the customer's face within the displayed image. FIG. 2 is a second diagram showing an example of the functional configuration of the image processing device. 1 is a diagram illustrating an example of a system configuration of a psychological data utilization system including an interest target estimation system.

Each embodiment will be described below with reference to the accompanying drawings. Note that, in this specification and the drawings, components having substantially the same functional configuration are designated by the same reference numerals, thereby omitting redundant explanation.

[First embodiment]
<System configuration of interest target estimation system>
First, the system configuration of the object of interest estimation system will be explained. The interest object estimation system is a system that estimates the object that a customer is interested in based on the customer's line of sight. In the first embodiment, a case will be described in which the object-of-interest estimation system is applied to a cosmetics store (for example, a dedicated booth) where beauty staff actually serve customers while applying makeup.

FIG. 1 is a first diagram showing an example of the system configuration of an interest target estimation system. As shown in FIG. 1, an object of interest estimation system 100 applied to a cosmetics store includes a dressing table 110, an imaging device 120, a line of sight sensor 130, and an image processing device 140.

The mirror stand 110 is a stand equipped with a mirror that reflects the customer's appearance as a mirror image when the customer 150 is seated at a predetermined position. It is assumed that the distance between the dressing table 110 and the customer 150 is approximately constant for all customers. In the example of FIG. 1, the beauty staff who applies makeup to the customer 150 is omitted, but in reality, it is assumed that there is a beauty staff behind or next to the customer 150.

The imaging device 120 is, for example, an RGB camera, and photographs the customer 150 at a predetermined frame rate (for example, 30 frames/second), and transmits the photographed image data to the image processing device 140. As a result, in the image processing device 140, even if the position of the face of the customer 150 moves while the beauty staff is applying makeup to the customer 150, the position coordinates of the face of the customer 150 within the image data (the imaging device 120) can be calculated sequentially.

Here, the positional relationship between the dressing table 110 and the imaging device 120 is fixed. Therefore, by sequentially calculating the positional coordinates of the face of the customer 150 in the image data (positional coordinates relative to the imaging device 120), the image processing device 140 can determine the position of the face of the customer 150 in the mirror table 110. Coordinates can be calculated sequentially. That is, according to the image processing device 140, the position of the face of the customer 150 in the mirror image can be sequentially identified.

In addition, although the example of FIG. 1 shows the case where the imaging device 120 is arranged at the upper part of the mirror stand 110, the position where the imaging device 120 is arranged is not limited to this. Further, in the example of FIG. 1, the imaging device 120 is integrally attached to the mirror stand 110, but it may be attached separately from the mirror stand 110. When attached separately, the imaging device 120 does not need to be placed on the front side of the customer 150 and may be placed on the back side of the customer 150.

The line of sight sensor 130 is, for example, an infrared camera, detects the line of sight of the customer 150 30 times per second, and transmits the detected line of sight data to the image processing device 140 . Thereby, the image processing device 140 can sequentially calculate the positional coordinates of the line of sight of the customer 150 (positional coordinates within the dressing table 110) while the beauty staff is applying makeup to the customer 150. That is, according to the image processing device 140, the position of the line of sight of the customer 150 in the mirror image can be sequentially acquired.

The image processing device 140 receives the image data transmitted from the imaging device 120, calculates and accumulates the position coordinates of the face of the customer 150 within the image data based on the received image data. Furthermore, the image processing device 140 calculates and accumulates the positional coordinates of the face of the customer 150 within the dressing table 110 based on the positional coordinates of the face of the customer 150 within the image data.

The image processing device 140 also receives the line-of-sight data transmitted from the line-of-sight sensor 130, calculates and accumulates the positional coordinates of the customer's 150 line of sight within the dressing table 110 based on the received line-of-sight data.

Furthermore, the image processing device 140 determines which of the customer's own face reflected in the mirror table 110 is based on the position coordinates of the customer's 150 face within the mirror table 110 and the position coordinates of the customer's 150's line of sight within the mirror table 110. The part of the face that the customer 150 is actually looking at is determined.

Further, the image processing device 140 estimates the part of the face that the customer 150 is interested in by counting the number of times the customer has viewed each part of the face.

<Photographing range and position coordinates of the imaging device>
Next, the photographing range of the imaging device 120 and the positional coordinates of the customer's face (positional coordinates within the image data, positional coordinates within the dressing table 110) will be explained. FIG. 2 is a first diagram for explaining the photographing range of the imaging device and the position coordinates of the customer's face within the image data and within the dressing table.

As shown in FIG. 2, the imaging device 120 placed on the mirror stand 110 photographs a photographing range 210 in the front direction of the mirror stand 110. In FIG. 2, a two-dot chain line 220 indicates the position where the customer 150 is seated.

Image data 230 is an example of image data obtained by photographing customer 150 with imaging device 120. As shown in FIG. 2, the coordinates of each pixel are defined in the image data 230 (for example, origin=coordinates (0,0)). Therefore, in the image processing device 140, by extracting the face area from the image data 230, the positional coordinates of the face of the customer 150 in the image data 230 and each part of the face of the customer 150 in the image data 230 are determined. The position coordinates of can be calculated.

Note that the position coordinates of the face of the customer 150 within the image data 230 may be, for example, the center position coordinates of the extracted face area (for example, a rectangular area), or the coordinates of the center position of the extracted face area (for example, a rectangular area). may be the position coordinates of each vertex of . Similarly, the position coordinates of each part of the face may be, for example, the center position coordinates of each divided area (for example, a rectangular area) obtained by dividing the extracted facial area (for example, a rectangular area) into a plurality of parts, or It may also be the position coordinates of each vertex of a segmented area (for example, a rectangular area).

Here, the image processing device 140 specifies a range 231 of the image data 230 that is reflected in the mirror table 110 based on the positional relationship between the mirror table 110 and the imaging device 120 and the photographing range 210 of the imaging device 120.

As a result, the image processing device 140 converts the positional coordinates of the face of the customer 150 in the image data 230 and the positional coordinates of each part of the face of the customer 150 in the image data 230 to the positional coordinates in the dressing table 110. Each can be converted to . As a result, the image processing device 140 can calculate the position coordinates of the face of the customer 150 within the mirror table 110 and the position coordinates of each part of the face of the customer 150 within the mirror table 110. In the example of FIG. 2, the coordinates are defined such that the lower left position of the dressing table 110 is the origin, the right direction as seen from the customer 150 is the X-axis direction, and the upward direction as seen from the customer 150 is the Y-axis direction.

Therefore, in the image processing device 140, the position coordinates of the face of the customer 150 in the image data 230 calculated based on the origin of the image data 230 are changed to the lower left position of the dressing table 110 (the lower right position of the range 231). Convert to the coordinates of the origin. In addition, the image processing device 140 calculates the positional coordinates of each part of the face of the customer 150 in the image data 230 calculated based on the origin of the image data 230 at the lower left position of the dressing table 110 (the lower right position of the range 231). ) into position coordinates with the origin as the origin.

As a result, the image processing device 140 calculates the positional coordinates of the face of the customer 150 within the dressing table 110 and the position coordinates of each part of the face of the customer 150 within the dressing table 110, with the lower left position of the dressing table 110 as the origin. can be calculated.

<Measurement range and position coordinates of the gaze sensor>
Next, the measurement range of the line-of-sight sensor 130 and the positional coordinates of the customer's 150 line of sight (positional coordinates within the dressing table 110) will be explained. FIG. 3 is a diagram for explaining the measurement range of the line-of-sight sensor and the position coordinates of the customer's line of sight within the dressing table.

As shown in FIG. 3, the line-of-sight sensor 130 disposed on the mirror stand 110 measures a measurement range 310 in the front direction of the mirror stand 110. In FIG. 3, a two-dot chain line 220 indicates the position where the customer 150 is seated.

When the line of sight 320 of the customer 150 is detected by the line of sight sensor 130, the image processing device 140 acquires the position of the line of sight of the customer 150 within the dressing table 110 from the detected line of sight data. In FIG. 3, a mark 330 indicates the acquired position of the customer's 150 line of sight within the dressing table 110.

As described above, the coordinates within the dressing table 110 are defined such that the lower left position is the origin, the right direction as seen from the customer 150 is the X-axis direction, and the upward direction as seen from the customer 150 is the Y-axis direction. Therefore, the image processing device 140 can calculate the positional coordinates of the line of sight of the customer 150 within the mirror table 110 by calculating the position coordinates of the mark 330 within the mirror table 110.

<Positional relationship between the position of the face and the position of the line of sight>
Next, the positional relationship between the position of the face of the customer 150 who is applying makeup and the position of the line of sight will be explained. FIG. 4 is a diagram for explaining the relationship between the positional coordinates of the customer's face within the dressing table and the positional coordinates of the customer's line of sight. Of these, FIG. 4(a) shows the appearance of the dressing table 110 at time=T0 (the positional relationship between the position of the face of the customer 150 and the position of the line of sight of the customer 150). The example in FIG. 4A shows that at time=T0, the line of sight of the customer 150 is located at the area below the eyes of the customer's own face reflected on the dressing table 110.

On the other hand, FIGS. 4(b) and 4(c) show the state of the dressing table 110 at time=T1 (the positional relationship between the position of the face of the customer 150 and the position of the line of sight of the customer 150). Note that the position of the line of sight within the dressing table 110 is the same in FIG. 4(b) and FIG. 4(c).

Here, FIG. 4(b) shows a case where the position of the face of the customer 150 does not move between time=T0 and time=T1. In the case of FIG. 4B, the line of sight of the customer 150 is shown to be around the ear of his or her own face reflected on the dressing table 110.

On the other hand, FIG. 4(c) shows a case where the position of the face of the customer 150 moves between time = T0 and time T1. In the case of FIG. 4C, the position of the line of sight of the customer 150 is shown to be around the bottom of the eyes of the customer's own face reflected on the dressing table 110.

In this way, the position of the face of the customer 150 reflected on the dressing table 110 during makeup is not constant; for example, when the customer 150 changes his posture, the position of the face reflected on the dressing table 110 also changes. Therefore, even if the position of the line of sight of the customer 150 within the dressing table 110 is the same, the part of the face that the customer 150 is looking at is not necessarily the same. Alternatively, even if the position of the customer's 150 line of sight within the dressing table 110 changes, the part of the face that the customer 150 is looking at does not necessarily change.

In other words, it is not possible to determine the part of the face that the customer 150 is actually looking at based only on the positional coordinates of the customer's 150 line of sight within the dressing table 110. Therefore, in determining the part of the face that the customer is looking at, the image processing device 140 uses the positional coordinates of the line of sight of the customer 150 within the dressing table 110 and the position of each part of the face of the customer 150 within the dressing table 110. Compare the coordinates (details will be explained later). Thereby, according to the image processing device 140, it is possible to accurately determine the part of the face that the customer 150 is actually looking at in the mirror image.

<Hardware configuration of image processing device>
Next, the hardware configuration of the image processing device 140 will be explained. FIG. 5 is a diagram illustrating an example of the hardware configuration of the image processing device. As shown in FIG. 5, the image processing device 140 includes a CPU (Central Processing Unit) 501, a ROM (Read Only Memory) 502, and a RAM (Random Access Memory) 503. The CPU 501, ROM 502, and RAM 503 form a so-called computer.

The image processing device 140 also includes an auxiliary storage section 504, an operation section 505, an I/F (Interface) section 506, and a drive section 507. Note that each piece of hardware in the image processing device 140 is interconnected via a bus 508.

The CPU 501 is a calculation device that executes various programs installed in the auxiliary storage unit 504 (for example, an image processing program described below).

ROM 502 is a nonvolatile memory. The ROM 502 functions as a main storage device that stores various programs, data, etc. necessary for the CPU 501 to execute the various programs installed in the auxiliary storage unit 504. Specifically, the ROM 502 functions as a main storage device that stores boot programs such as BIOS (Basic Input/Output System) and EFI (Extensible Firmware Interface).

The RAM 503 is a volatile memory such as DRAM (Dynamic Random Access Memory) or SRAM (Static Random Access Memory). The RAM 503 functions as a main storage device that provides a work area in which various programs installed in the auxiliary storage unit 504 are expanded when executed by the CPU 501.

The auxiliary storage unit 504 is an auxiliary storage device that stores various programs and information obtained by executing the various programs. A data storage unit, which will be described later, is realized in the auxiliary storage unit 504.

The operation unit 505 is an input device through which an administrator of the image processing apparatus 140 inputs various instructions to the image processing apparatus 140. The I/F unit 506 is a communication device that is connected to the imaging device 120 and the line-of-sight sensor 130 and allows the image processing device 140 to receive image data and line-of-sight data from the imaging device 120 and the line-of-sight sensor 130.

The drive section 507 is a device for setting the recording medium 510. The recording medium 510 herein includes a medium for recording information optically, electrically, or magnetically, such as a CD-ROM, a flexible disk, or a magneto-optical disk. Further, the recording medium 510 may include a semiconductor memory or the like that electrically records information, such as a ROM or a flash memory.

The various programs to be installed in the auxiliary storage unit 504 are installed by, for example, setting the distributed recording medium 510 in the drive unit 507 and reading out the various programs recorded on the recording medium 510 by the drive unit 507. be done. Alternatively, various programs installed in the auxiliary storage unit 504 may be installed by being downloaded from a network (not shown).

<Functional configuration of image processing device>
Next, the functional configuration of the image processing device 140 will be explained. FIG. 6 is a first diagram showing an example of the functional configuration of the image processing device. As described above, an image processing program is installed in the image processing device 140, and by executing the program, the image processing device 140 stores the face position coordinates in the image data in the face position coordinate storage unit 610, the face position coordinates in the dressing table, etc. It functions as a storage section 620. Further, the image processing device 140 functions as an in-dressing line-of-sight position coordinate storage section 630 and a parts determination section 640. Furthermore, the image processing device 140 functions as a frequency calculation section 650, a frequency increase determination section 660, an increase amount calculation section 670, an object of interest estimation section 680, and an output section 690.

The face position coordinate storage unit 610 in image data receives the image data transmitted from the imaging device 120, and extracts the face area of the customer 150 from the received image data. In addition, the in-image data face position coordinate storage unit 610 specifies the position of the extracted face area in each image data, thereby identifying the position coordinates (x coordinate, y coordinate). Note that, hereinafter, the position coordinates of the face of the customer 150 within the image data will be referred to as face position coordinates within the image data.

Further, the image data face position coordinate storage unit 610 stores the calculated face position coordinates in the image data in the data storage unit 600 in association with time information, and notifies the face position coordinate storage unit 620 in the dressing table.

When the face position coordinate storage unit 620 in the mirror table is notified of the face position coordinates in the image data from the image data face position coordinate storage unit 610, the face position coordinate storage unit 620 stores the position coordinates (X coordinates, Y coordinates, Y coordinates, coordinates). Note that, hereinafter, the position coordinates of the face of the customer 150 within the dressing table 110 will be referred to as face position coordinates within the dressing table.

In addition, the face position coordinate storage unit 620 stores the calculated face position coordinates in the mirror table in the data storage unit 600 in association with time information.

The mirror table line-of-sight position coordinate storage unit 630 is an example of an acquisition unit. The in-dressing table line-of-sight position coordinate storage unit 630 receives the line-of-sight data transmitted from the line-of-sight sensor 130, and stores the position coordinates (X coordinate, Y coordinate) of the customer's 150 line of sight in the dressing table 110 based on the received line-of-sight data. Calculate. Note that, hereinafter, the positional coordinates of the line of sight of the customer 150 within the dressing table 110 will be referred to as the line-of-sight position coordinates within the dressing table.

Further, the mirror table line-of-sight position coordinate storage section 630 stores the calculated table-line line-of-sight position coordinates in the data storage section 600 in association with time information.

The parts determining section 640 is an example of a determining section. The parts determining unit 640 reads out the line-of-sight position coordinates in the dressing table and the face position coordinates in the dressing table at each time, which are stored in the data storage unit 600. In addition, when the face area of the customer 150 is divided into a plurality of areas based on the read face position coordinates in the dressing table, the parts determination unit 640 determines the position coordinates (X coordinate, Y coordinate, coordinates).

Furthermore, the parts determining unit 640 determines which of the calculated positional coordinates within the mirror stand 110 of each segmented area includes the read line-of-sight position coordinate within the mirror stand. Furthermore, when determining that the line-of-sight position coordinates within the dressing table are included in the position coordinates of any of the segmented areas, the parts determination unit 640 associates information indicating the segmented area (referred to as facial parts information) with time information. The data is stored in the data storage unit 600.

Thereby, the parts determining unit 640 can determine which part of the customer's 150's own face reflected in the dressing table 110 is viewed at each time.

The number calculation unit 650 calculates the number of times the customer 150 looked at each part of the face by dividing and totaling the number of times the customer 150 looked at each part of the face at each time, which is stored in the data storage unit 600. do.

The frequency increase determination unit 660 further divides and totals the number of times the customer 150 views each body part for each predetermined time range. Thereby, the number increase amount determining unit 660 can determine how the number of times the customer 150 has viewed each part has changed as time progresses.

The increase calculation unit 670 divides and totals the number of times the customer 150 has looked (the number of times he has looked at any part of his own face) for all body parts, for each predetermined time range. Thereby, the increase calculation unit 670 can determine how the number of times the customer 150 has looked at any part of his or her own face has changed as time passes.

The object of interest estimating section 680 estimates the object of interest of the customer 150 based on the aggregation result by the frequency calculation section 650, the aggregation result by the frequency increase determination section 660, and the aggregation result by the increase amount calculation section 670. Specifically, the object of interest estimation unit 680 determines whether the customer 150 is interested in any part of his or her own face, and if it is determined that the customer 150 is interested, which part of the customer's face the customer 150 is interested in. Estimate.

The output unit 690 outputs the estimation result regarding the object of interest of the customer 150 estimated by the object of interest estimation unit 680. The output unit 690 outputs the information to, for example, a beauty staff member who is applying makeup to the customer 150. Thereby, the beauty staff can provide customer service to the customer 150 according to the estimation result.

<Example of data in data storage section>
Next, an example of data stored in the data storage unit 600 will be explained. FIG. 7 is a diagram illustrating an example of data stored in the data storage unit. As shown in FIG. 7, the data 700 stored in the data storage unit 600 includes data items such as "time information", "line of sight position coordinates in mirror table", "face position coordinates in image data", and "inside mirror table". Contains "face position coordinates" and "face parts information".

In the "time information", the time at which the line-of-sight position coordinates in the dressing table, the face position coordinates in the image data, the face position coordinates in the dressing table, and the facial part information are stored in the data storage unit 600 is recorded.

In the “mirror table line-of-sight position coordinates”, the mirror table line-of-sight position coordinates (X coordinate, Y coordinate) calculated by the mirror table line-of-sight position coordinate storage unit 630 are recorded.

In the “image data face position coordinates” field, the image data face position coordinates (x coordinates, y coordinates) calculated by the image data face position coordinate storage unit 610 are recorded.

In the “face position coordinates in the mirror table”, the face position coordinates (X coordinate, Y coordinate) in the mirror table calculated by the face position coordinate storage unit 620 in the mirror table are recorded.

Facial parts information determined by the parts determining unit 640 is recorded in “facial parts information”. Note that FIG. 8 is a diagram for explaining facial parts. As described above, when the face area 800 of the customer 150 is divided into a plurality of areas, the parts determination unit 640 determines the positional coordinates (X coordinates, Y coordinates) of each divided area within the mirror stand 110. It is calculated as the position coordinates of each part of the face of the customer 150. Note that the parts determining unit 640 calculates the positional coordinates of each part of the face of the customer 150 within the dressing table 110 based on the face position coordinates within the dressing table.

FIG. 8 shows each divided area (divided areas 801 to 805 and divided areas 811 to 815) when the face area 800 of the customer 150 is divided into ten areas. According to the example of FIG. 8, information indicating one of the segmented areas 801 to 805 and segmented areas 811 to 815 is recorded as the facial part.

<Flow of interest target estimation process>
Next, the flow of region of interest estimation processing by the image processing device 140 will be explained. 9 and 10 are first and second flowcharts showing the flow of the interest target estimation process. When the customer 150 is seated at a predetermined position with respect to the dressing table 110, the object of interest estimation process shown in FIGS. 9 and 10 is started.

In step S901, the image data face position coordinate storage unit 610 receives image data from the imaging device 120, extracts the face area of the customer 150, calculates the face position coordinates in the image data, and stores the image data in the data storage unit 600. Accumulate in.

In step S902, the in-dressing face position coordinate storage unit 620 calculates the in-dressing face position coordinates based on the face position coordinates in the image data calculated by the face position coordinates in image data storage unit 610, and Accumulate in.

In step S903, the in-dressing line-of-sight position coordinate storage unit 630 receives the line-of-sight data from the line-of-sight sensor 130, calculates the line-of-sight position coordinate in the dressing table, and stores it in the data storage unit 600.

In step S904, the parts determination unit 640 determines each part of the customer's 150 face (for example, segmented areas 801 to 805, segmented area 811 ~815) within the mirror table 110.

In step S905, the parts determination unit 640 compares the calculated positional coordinates of each part of the face of the customer 150 within the dressing table 110 with the line-of-sight position coordinates within the dressing table, thereby determining whether the part determining unit 640 is able to determine whether the face of the face including the line-of-sight position coordinates within the dressing table is Determine the part of the body.

In step S906, parts determining section 640 stores the determined facial parts in data storage section 600 as facial parts information.

In step S907, the number calculation unit 650 determines whether a predetermined time has elapsed since the previous aggregation timing. If it is determined in step S907 that the predetermined time has elapsed (in the case of Yes in step S907), the process advances to step S1001 in FIG. 10. On the other hand, if it is determined in step S907 that the predetermined time has not elapsed (No in step S907), the process advances to step S908.

In step S1001 in FIG. 10, the number calculation unit 650 totals the number of times the customer 150 has viewed each part of the face based on the data 700 accumulated in the data storage unit 600.

In step S1002, the number increase determination unit 660 divides and totals the number of times the customer 150 looks at each part of the face for each predetermined time range, and as time progresses, the number of times the customer 150 looks at each part of the face. Determine how the number of views has changed. In addition, the increase calculation unit 670 aggregates the number of times the customer 150 has looked at any part of the face, divided into predetermined time ranges, and as time progresses, the number of times the customer 150 has looked at any part of the face. Determine how the number of times viewed has changed.

In step S1003, the object of interest estimating unit 680 estimates the object of interest of the customer 150 based on the aggregation results by the frequency calculation unit 650, the aggregation results by the frequency increase determination unit 660, and the aggregation results by the increase amount calculation unit 670. do.

In step S1004, the output unit 690 outputs the estimation result regarding the object of interest of the customer 150, and returns to step S908 in FIG. 9.

In step S908, the image data face position coordinate storage unit 610 determines whether or not to end the object of interest estimation process. If it is determined in step S908 that the process does not end (No in step S908), the process returns to step S901. On the other hand, if it is determined in step S908 to end (Yes in step S908), the region of interest estimation process ends.

<Example of aggregated results>
Next, the results of aggregation of data accumulated in the data storage unit 600 will be explained. FIG. 11 is a diagram showing an example of the total results. Among them, FIG. 11A is a diagram showing an example of the total results by the number of times calculation unit 650. In FIG. 11(a), the horizontal axis is the part name (for example, the names representing the segmented areas 801 to 805 and the segmented areas 811 to 815), and the vertical axis indicates the number of times the customer 150 has viewed it. As shown in FIG. 11A, when a customer 150 looks at his own face reflected on the dressing table 110, the number of times he looks at each part is different, and there is variation between parts.

On the other hand, FIG. 11(b) is a diagram illustrating an example of a total result by the frequency increase amount determination unit 660. As shown in FIG. 11(b), when a customer 150 looks at his own face reflected on the dressing table 110, there are areas in which his interest increases and areas in which he becomes less interested as time passes.

Therefore, the object of interest estimating section 680 can estimate the current object of interest of the customer 150, for example, based on the aggregation result by the frequency calculation section 650 and the aggregation result by the frequency increase amount determination section 660.

As is clear from the above description, the image processing device 140 according to the first embodiment acquires the position of the customer's line of sight directed toward the customer's mirror image. Furthermore, the image processing device 140 according to the first embodiment identifies the position of the customer's face in the mirror image, and determines the part of the customer's face in the mirror image that corresponds to the specified line of sight position. .

Thereby, according to the image processing device 140 according to the first embodiment, it is possible to accurately determine the part of the face that the customer is looking at in the mirror image.

[Second embodiment]
In the first embodiment, a case has been described in which the part of the customer's face in the mirror image corresponding to the position of the customer's line of sight directed toward the customer's mirror image is determined. On the other hand, in the second embodiment, the position of the customer's line of sight directed toward the displayed image of the customer captured by the imaging device is acquired, and the customer in the displayed image corresponding to the acquired line of sight position is A case will be explained in which the facial part of the person is determined. The second embodiment will be described below, focusing on the differences from the first embodiment.

<System configuration of interest target estimation system>
First, the system configuration of the object of interest estimation system according to the second embodiment will be described. FIG. 12 is a second diagram showing an example of the system configuration of the object of interest estimation system. The difference from the system of interest estimation system 100 shown in FIG. This is different from the function of the device 140.

The display device 1210 receives, from the image processing device 1220, image data captured by the imaging device 120, which has been horizontally reversed by the image processing device 1220, and displays the image data.

The image processing device 1220 receives image data transmitted from the imaging device 120, horizontally inverts the received image data, and transmits the image data to the display device 1210. Further, the image processing device 1220 extracts a face area from the horizontally reversed image data, and sequentially calculates the position coordinates (X coordinates, Y coordinates) of the face of the customer 150 within the display device 1210 (the 150 faces are sequentially identified).

The image processing device 1220 also receives the line-of-sight data transmitted from the line-of-sight sensor 130, and sequentially calculates the position coordinates of the line of sight of the customer 150 within the display device 1210 based on the received line-of-sight data (within the display image). 150).

The image processing device 1220 also processes the image of the customer 150 displayed on the display device 1210 based on the positional coordinates of the customer's 150 face within the display device 1210 and the positional coordinates of the customer's 150's line of sight within the display device 1210. The part of the face that the customer 150 is actually looking at is determined.

Furthermore, the image processing device 1220 estimates the parts of the customer's 150 that are of interest by counting the number of times the customer has viewed each part of the face.

<Photographing range and position coordinates of the imaging device>
Next, the photographing range of the imaging device 120 and the positional coordinates of the customer's face (positional coordinates within the display device 1210) will be described. FIG. 13 is a diagram for explaining the photographing range of the imaging device and the position coordinates of the customer's face within the display device.

As shown in FIG. 13, the imaging device 120 disposed on the display device 1210 photographs a photographing range 210 in the front direction of the display device 1210. In FIG. 13, a two-dot chain line 220 indicates the position where the customer 150 is seated.

Image data 230 is an example of image data obtained by photographing customer 150 with imaging device 120. Further, the image data 1310 is image data obtained by horizontally inverting the image data 230 and extracting a range 231 to be displayed on the display device 1210.

As shown in FIG. 13, the coordinates of each pixel when displayed on the display device 1210 are defined in the image data 1310. In the example of FIG. 13, the coordinates are defined with the lower left of the display device 1210 as the origin, the right direction as seen from the customer 150 as the X-axis direction, and the upward direction as seen from the customer 150 as the Y-axis direction.

The image processing device 1220 calculates the position coordinates of the face of the customer 150 in the image data 1310 and the position of each part of the face in the image data 1310 based on the coordinates (0, 0) of the origin of the image data 1310. Calculate coordinates.

Note that in the case of the second embodiment, the position coordinates of the face of the customer 150 within the image data 1310 are equal to the position coordinates of the face of the customer 150 within the display device 1210. Further, the positional coordinates of each part of the face of the customer 150 within the image data 1310 are equal to the positional coordinates of each part of the face of the customer 150 within the display device 1210.

<Functional configuration of image processing device>
Next, the functional configuration of the image processing device 1220 will be explained. FIG. 14 is a second diagram showing an example of the functional configuration of the image processing device. Differences from the functional configuration shown in FIG. 6 include an image data acquisition section 1410, an image data correction section 1420, a corrected image data display section 1430, a face position coordinate storage section 1440 in the display device, and a line-of-sight position coordinate storage section in the display device. 1450 and a parts determining section 1460.

Image data acquisition section 1410 receives image data transmitted from imaging device 120, and notifies image data correction section 1420 of the received image data.

The image data correction unit 1420 extracts a range to be displayed on the display device 1210 from the notified image data, and corrects the extracted image data by horizontally reversing it. Further, the image data correction unit 1420 notifies the corrected image data display unit 1430 of the corrected image data.

The corrected image data display section 1430 transmits the corrected image data notified from the image data correction section 1420 to the display device 1210. Further, the corrected image data display section 1430 notifies the display device face position coordinate storage section 1440 of the corrected image data notified from the image data correction section 1420.

When the face position coordinate storage unit 1440 in the display device is notified of the corrected image data from the corrected image data display unit 1430, it extracts the face area and stores the position coordinates of the face of the customer 150 in the display device 1210. (X coordinate, Y coordinate) is calculated. Note that, hereinafter, the position coordinates of the face of the customer 150 within the display device 1210 will be referred to as the face position coordinates within the display device.

Further, the display device face position coordinate storage unit 1440 stores the calculated display device face position coordinates in the data storage unit 600 in association with time information.

The in-display device line-of-sight position coordinate storage unit 1450 receives the line-of-sight data transmitted from the line-of-sight sensor 130, and stores the line-of-sight position coordinates (X coordinate, Y coordinates). Note that, hereinafter, the positional coordinates of the line of sight of the customer 150 within the display device 1210 will be referred to as the line-of-sight position coordinates within the display device.

Further, the display device line-of-sight position coordinate storage unit 1450 stores the calculated line-of-sight position coordinates in the display device in the data storage unit 600 in association with time information.

The parts determining unit 1460 reads out the line-of-sight position coordinates in the display device and the face position coordinates in the display device at each time, which are stored in the data storage unit 600. In addition, when the face area of the customer 150 is divided into a plurality of areas based on the read face position coordinates within the display device, the parts determination unit 640 determines the position coordinates (X coordinates) of each divided area within the display device 1210. , Y coordinate).

Furthermore, the parts determining unit 1460 determines which of the calculated positional coordinates within the display device 1210 of each segmented region includes the read line-of-sight position coordinate within the display device. Furthermore, when determining that the line-of-sight position coordinates within the display device are included in the position coordinates of any segmented area, the parts determination unit 1460 associates information indicating the segmented area (facial parts information) with time information. The data is stored in the data storage unit 600.

Thereby, parts determining section 1460 can determine which part of customer's 150's own face displayed in display device 1210 has looked at each time.

As is clear from the above description, the image processing device 1220 according to the second embodiment acquires the position of the customer's line of sight directed toward the displayed image of the customer captured by the imaging device. Furthermore, by identifying the position of the customer's face within the display device, the image processing device 1220 according to the second embodiment determines the part of the customer's face within the display image that corresponds to the identified line of sight position. judge.

Thereby, according to the image processing device 1220 according to the second embodiment, it is possible to accurately determine the part of the face that the customer is looking at in the displayed image.

[Third embodiment]
In the first and second embodiments described above, the estimation results regarding the object of interest of the customer 150 have been described as being output to the beauty staff. However, the output destination of the estimation results regarding the target of interest of the customer 150 is not limited to this, and may be used for other purposes. The third embodiment will be described below.

FIG. 15 is a diagram illustrating an example of a system configuration of a psychological data utilization system including an interest target estimation system. As shown in FIG. 15, the psychological data utilization system 1500 includes a plurality of image processing devices 140, an analysis server 1510, analysis devices 1520 and 1530, and a mobile terminal 1540.

Each of the plurality of image processing devices 140 sends the estimation results regarding the customer's interest object to the analysis server 1510 in association with the customer information.

The analysis server 1510 receives the estimation results and customer information regarding the object of interest of the customer from each of the plurality of image processing devices 140, and stores them in the analysis data storage unit 1512 as analysis data.

The analysis server 1510 also includes a psychological data analysis unit 1511, which reads out the estimation results and customer information regarding the target of interest of the customer from the analysis data storage unit 1512, and collects the psychological data of each customer. Analyze. Furthermore, the psychological data analysis unit 1511 transmits the analysis results to, for example, an analysis device 1520 in the product sales department and an analysis device 1530 in the product development department.

Furthermore, the psychological data analysis unit 1511 transmits recommended products derived for each customer according to the analysis results to the customer's mobile terminal 1540.

When the analysis device 1520 of the product sales department receives the analysis results from the analysis server 1510, the employees of the product sales department formulate a product sales strategy based on the received analysis results. Note that the planned product sales strategy is fed back to the cosmetics store.

Further, when the analysis device 1530 of the product development department receives the analysis result from the analysis server 1510, the employee of the product development department develops the product based on the received analysis result. The results of product development are fed back to the product sales department.

Mobile terminal 1540 receives recommended products from analysis server 1510 and displays them. This allows each customer to receive product recommendations based on their own psychological data.

In this way, according to the psychological data utilization system 1500, by analyzing the estimation results regarding the target of interest of the customer 150 as the psychological data of the customer 150, the estimation results can be utilized for various purposes.

[Other embodiments]
In the first embodiment described above, the position coordinates of the face of the customer 150 within the dressing table 110 are calculated based on the image data photographed by the imaging device 120. However, the method for calculating the position coordinates of the face of the customer 150 is not limited to this, and for example, the position coordinates of the face of the customer 150 may be calculated based on the line-of-sight data detected by the line-of-sight sensor 130. In this case, there is an advantage that there is no need to arrange the imaging device 120.

Furthermore, in each of the above embodiments, the part of the face that the customer is looking at is determined by dividing the facial area into multiple areas and determining which of the divided areas the position coordinates of the customer's line of sight are included in. did. However, the method for determining the part of the face that the customer is looking at is not limited to this. For example, by extracting a region consisting of the shape of the parts themselves included in the facial region (eyes, nose, mouth, etc.) and determining which shape region contains the positional coordinates of the customer's line of sight, it is possible to It may also be possible to determine which part of the body is affected. In other words, the part that the customer is looking at may be any segmented area of the facial area or any shape area of the facial area.

Further, in each of the above embodiments, the face area is extracted as a rectangular area, but the shape of the extracted face area is not limited to this. Similarly, in each of the above embodiments, the segmented area is a rectangular area, but the shape of the segmented area is not limited to this. Further, the method of dividing the face area and the number of divisions are not limited to the example shown in FIG. 8.

Further, in each of the embodiments described above, the case where the object of interest estimation system is applied to a cosmetics store has been described, but it may be applied to a store that provides other products or services. Further, the application destination is not limited to a store, but may also be applied at home. In this case, the estimation of the object of interest will be performed not for the customer of the store but for the user who uses the object of interest estimation system.

Note that, in the disclosed technology, forms such as those described below are possible.
(Additional note 1)
Obtaining the position of the user's line of sight directed toward a mirror image of the user or a display image of the user captured by an imaging device;
a position of the user's face in the mirror image, or a part of the user's face in the mirror image that corresponds to the position of the line of sight obtained by identifying the position of the user's face in the display image; , or determining a part of the user's face within the displayed image;
An image processing program that causes a computer to perform processing.
(Additional note 2)
The image processing program according to supplementary note 1, which specifies the position of the user's face in the mirror image based on the position of the user's face in the image captured by an imaging device.
(Additional note 3)
The position of the user's line of sight in the mirror image is determined by determining which segmented area of the user's face area in the mirror image is included in a plurality of areas. The image processing program according to supplementary note 2, which determines a part of a user's face.
(Additional note 4)
The position of the user's line of sight in the display image is determined in which divided area of the user's face area in the display image is divided into a plurality of areas. The image processing program according to Supplementary Note 1, wherein the image processing program determines a part of the user's face within the image.
(Appendix 5)
an acquisition unit that acquires the position of the user's line of sight directed toward a mirror image of the user or a display image of the user captured by an imaging device;
a position of the user's face in the mirror image, or a part of the user's face in the mirror image that corresponds to the position of the line of sight obtained by identifying the position of the user's face in the display image; or a determination unit that determines a part of the user's face within the display image.
(Appendix 6)
Obtaining the position of the user's line of sight directed toward a mirror image of the user or a display image of the user captured by an imaging device;
a position of the user's face in the mirror image, or a part of the user's face in the mirror image that corresponds to the position of the line of sight obtained by identifying the position of the user's face in the display image; , or determining a part of the user's face within the displayed image;
An image processing method in which processing is performed by a computer.

Note that the present invention is not limited to the configurations shown here, such as combinations of other elements with the configurations listed in the above embodiments. These points can be modified without departing from the spirit of the present invention, and can be appropriately determined depending on the application thereof.

100: Interest object estimation system 110: Mirror table 120: Imaging device 130: Gaze sensor 140: Image processing device 610: Image data face position coordinate storage section 620: Mirror table face position coordinate storage section 630: Mirror table gaze position coordinate storage section 640: Parts determination section 650: Number of times calculation section 660: Number of times increase amount determination section 670: Increase amount calculation section 680: Interest object estimation section 690: Output section 1200: Interest object estimation system 1210: Display device 1220: Image processing device 1410: Image data acquisition unit 1420 : Image data correction unit 1430 : Post-correction image data display unit 1440 : Display device face position coordinate storage unit 1450 : Display device internal line of sight position coordinate storage unit 1460 : Parts determination unit

Claims (6)

Obtaining the position of the user's line of sight directed toward a mirror image of the user or a display image of the user captured by an imaging device;
a position of the user's face in the mirror image, or a part of the user's face in the mirror image that corresponds to the position of the line of sight obtained by identifying the position of the user's face in the display image; , or determining a part of the user's face in the displayed image;
estimating the current object of interest of the user by determining a temporal change in the number of times the user looks at each part of the face based on the determined result;
An image processing program that causes a computer to perform processing. The image processing program according to claim 1, wherein the position of the user's face in the mirror image is specified based on the position of the user's face in an image photographed by an imaging device. By determining which segmented area of the user's face area in the mirror image is included in a plurality of areas, the position of the user's line of sight in the mirror image is included. The image processing program according to claim 2, which determines a part of a user's face. The position of the user's line of sight in the display image is determined in which segmented area of the user's face area in the display image is segmented into a plurality of regions, The image processing program according to claim 1, wherein the image processing program determines a part of the user's face within the image. an acquisition unit that acquires the position of the user's line of sight directed toward a mirror image of the user or a display image of the user captured by an imaging device;
a position of the user's face in the mirror image, or a part of the user's face in the mirror image that corresponds to the position of the line of sight obtained by identifying the position of the user's face in the display image; or a determination unit that determines a part of the user's face within the display image;
An image processing device comprising: an estimating unit that estimates a current target of interest of the user by determining a temporal change in the number of times the user looks at each part of the face based on the determined result. Obtaining the position of the user's line of sight directed toward a mirror image of the user or a display image of the user captured by an imaging device;
a position of the user's face in the mirror image, or a part of the user's face in the mirror image that corresponds to the position of the line of sight obtained by identifying the position of the user's face in the display image; , or determining a part of the user's face in the displayed image;
estimating the current object of interest of the user by determining a temporal change in the number of times the user looks at each part of the face based on the determined result;
An image processing method in which processing is performed by a computer.

Images