JP7673028B2 - Information processing device and information processing method - Google Patents

Description

The present invention relates to an information processing device and an information processing method.

Conventionally, there is known a conference system that realizes a web conference among multiple participants by transmitting images captured by the terminals of multiple participants who participate in the conference remotely and collected audio (see, for example, Patent Document 1).

JP 2011-205243 A

In a web conference, if the monitor of a participant's terminal is located far away from the imaging device, the participant often looks at the monitor without looking at the imaging device. In this case, the participant's eyes do not meet those of the other participants on the other participants' terminals, which creates a problem of impediment to non-verbal communication that would be achieved by making eye contact.

Therefore, the present invention has been made in consideration of these points, and aims to enable participants to align their gaze with that of other participants.

The information processing device according to the first aspect of the present invention includes an acquisition unit that acquires a plurality of images of a subject's face captured by an imaging device, a size determination unit that acquires size information indicating the size of the subject's eyeballs and the size of the iris, thereby determining the size of the eyeballs and the size of the iris, an estimation unit that estimates the ratio of the width of the iris to the width of the white of the eye when the subject faces forward by analyzing a first image acquired by the acquisition unit, and an angle determination unit that analyzes a second image acquired after the acquisition unit acquires the first image to determine a lateral shift angle, which is the angle at which the face shown in the second image is shifted laterally when the front direction of the subject's face is used as a reference, and a method of determining the size of the eyeballs and the size of the iris. The device has a width specification unit that specifies a ratio between the width of the subject's iris, the left white of the eye width indicating the width of the left white of the eye that is the white of the eye reflected to the left of the iris, and the right white of the eye width indicating the width of the right white of the eye that is the white of the eye reflected to the right of the iris when the subject looks at the front direction in the second image with the face shifted by the lateral shift angle with respect to the front direction in the second image, based on the size, the lateral shift angle, and the estimated ratio; a correction unit that corrects the second image so that the shape of the iris and the white of the eye of the subject included in the second image corresponds to the specified ratio of the width of the iris, the width of the left white of the eye, and the width of the right white of the eye; and an output unit that outputs the second image in which the subject's eyes have been corrected by the correction unit.

The estimation unit may estimate the ratios corresponding to each of the subject's left and right eyes by analyzing the first image acquired by the acquisition unit.
The estimation unit may analyze each of the multiple first images acquired by the acquisition unit to determine the ratio between the width of the pupil and the width of the white of the eye of the subject shown in each of the multiple first images, and estimate the ratio between the width of the pupil and the width of the white of the eye when the subject is facing forward by averaging the ratios determined for each of the multiple first images.

The angle identification unit may analyze the first image to identify the lateral shift angle of the face of the subject captured in the first image relative to the front direction of the face of the subject, and the estimation unit may estimate the ratio of the width of the pupil of the subject's eye to the width of the white of the eye of the subject captured in each of the first images by analyzing each of the first images when the angle identification unit identifies a different lateral shift angle.

The estimation unit may estimate the ratio between the width of the subject's pupil and the width of the white of the eye in the first image acquired by the acquisition unit, on the condition that the white of the eye is shown on both sides of the pupil of the subject in the first image acquired by the acquisition unit.

The information processing device may further have a color identification unit that identifies the color of the iris, the color of the left white of the eye, and the color of the right white of the eye by analyzing the first image, and the correction unit may correct the color of the subject's eyes shown in the second image based on the color of the iris, the color of the left white of the eye, and the color of the right white of the eye identified by the color identification unit.

The information processing device may further include a iris analysis unit that identifies the size of the light reflected in the iris and the position of the light in the iris by analyzing the first image, and the correction unit may depict the light relative to the iris based on the identified size of the light and the identified position of the light.

The information processing device may further include a iris analysis unit that determines the ratio between the width of the iris and the width of the pupil in the iris by analyzing the first image, and the correction unit may depict the pupil relative to the iris based on the determined ratio between the width of the iris and the width of the pupil in the iris.

The angle identification unit may analyze the second image to identify a vertical deviation angle, which is an angle at which the face of the subject reflected in the second image is shifted in the up-down direction when the front direction of the face of the subject is used as a reference, and the correction unit may correct the eyes of the subject reflected in the second image when the magnitude of the vertical deviation angle is equal to or smaller than a predetermined angle.

An information processing method according to a second aspect of the present invention includes the steps of acquiring a plurality of images of a subject's face captured by an imaging device, executed by a computer; identifying the size of the subject's eyeballs and the size of the iris by acquiring size information indicating the size of the eyeballs and the size of the iris; estimating the ratio of the width of the iris to the width of the white of the eye when the subject faces forward by analyzing the acquired first image; identifying the lateral deviation angle, which is the angle at which the face shown in the second image is shifted laterally when the front direction of the subject's face is used as a reference by analyzing a second image acquired after the first image is acquired; and determining the identified eyeball size, front eye size, and iris size. The method includes a step of determining a ratio between the width of the subject's pupil when the subject looks toward the front in the second image with the face shifted by the lateral shift angle relative to the front direction based on the size of the pupil, the left white of the eye width indicating the width of the left white of the eye that is the white of the eye reflected to the left of the pupil, and the right white of the eye width indicating the width of the right white of the eye that is the white of the eye reflected to the right of the pupil, based on the estimated ratio and the lateral shift angle; a step of correcting the second image so that the shape of the pupil and the white of the eye of the subject included in the second image corresponds to the determined ratio of the width of the pupil, the width of the left white of the eye, and the width of the right white of the eye; and a step of outputting the second image in which the subject's eyes have been corrected.

The present invention allows participants to align their gaze with that of other participants.

FIG. 1 is a diagram illustrating an overview of an information processing device. FIG. 2 is a diagram illustrating a functional configuration of an information processing device. 13A and 13B are diagrams illustrating the state of the eyeballs when the subject's face is shifted to the side; FIG. 4 is a close-up view of the eyeball of the subject shown in FIG. 3. 5 is an enlarged view of a sector portion of the eyeball shown in FIG. 4 that connects the center of the eyeball, the left end of the left white of the eye width, and the right end of the right white of the eye width. 13A and 13B are diagrams showing the relationship between the left white of the eye width in real space and the left white of the eye width shown in the second image, and the relationship between the right white of the eye width in real space and the right white of the eye width shown in the second image. FIG. 13 is a diagram illustrating an example of correction of a subject's eyes. 4 is a flowchart showing a process flow in an information processing device.

[Overview of information processing device 1]
1 is a diagram showing an overview of an information processing device 1. The information processing device 1 is a computer that corrects the eyes of a subject appearing in an image captured by an imaging device. The information processing device 1 is communicably connected to a first terminal 2 and a second terminal 3 via a communication network such as the Internet or a wireless LAN (Local Area Network). The information processing device 1 has a function for realizing a Web conference, for example, and outputs an image of a subject captured at the first terminal 2 and a sound collected at the first terminal 2 to the second terminal 3, and outputs an image of a subject captured at the second terminal 3 and a sound collected at the second terminal 3 to the first terminal 2.

The first terminal 2 and the second terminal 3 are, for example, terminals such as personal computers or smartphones having an imaging device and a microphone, and transmit images captured by the imaging device and sounds collected by the microphone to the information processing device 1. The first terminal 2 and the second terminal 3 may acquire images captured by an imaging device connected to them and transmit them to the information processing device 1. In the following explanation, an example of correcting the eyes of a subject appearing in an image captured by the first terminal 2 will be explained. The information processing device 1 may correct the eyes of a subject appearing in an image captured by the second terminal 3. Also, although only one second terminal 3 is shown in FIG. 1, there may be multiple second terminals.

The information processing device 1 acquires multiple first images of the subject's face from the first terminal 2 ((1) in FIG. 1). The information processing device 1 analyzes the multiple acquired first images to estimate the ratio between the pupil width, which is the width of the pupil when the subject faces forward, and the white of the eye width, which is the width of the white of the eye when the subject faces forward ((2) in FIG. 1). The pupil width indicates the width (diameter) of the circle indicating the pupil, and the white of the eye width indicates the sum of the maximum widths of the two white of the eye areas on the left and right of the pupil.

After estimating the ratio between the width of the pupil and the width of the white of the eye, the information processing device 1 acquires a second image from the first terminal 2 ((3) in FIG. 1). By analyzing the acquired second image, the information processing device 1 identifies the lateral deviation angle, which is the angle at which the face shown in the second image is shifted laterally when the front direction of the subject's face is used as a reference ((4) in FIG. 1).

Based on the previously acquired size of the subject's eyeball and pupil, the identified lateral deviation angle, and the estimated ratio of pupil width to white width, the information processing device 1 identifies the ratio of the pupil width, left white width, and right white width of the subject when the subject looks straight ahead in a state in which the subject's face is shifted by the lateral deviation angle with respect to the front direction in the second image ((5) in FIG. 1). The left white width refers to the maximum width of the left white area, which is the white of the eye reflected to the left of the pupil, and the right white width refers to the maximum width of the right white area, which is the white of the eye reflected to the right of the pupil.

The information processing device 1 corrects the eyes of the subject shown in the second image so that the shapes of the pupils and whites of the subject's eyes in the second image correspond to the specified ratio of pupil width, left white width, and right white width ((6) in FIG. 1). The information processing device 1 outputs the corrected second image, which is the corrected second image, to the second terminal 3 ((7) in FIG. 1). In this way, the information processing device 1 can align the gaze of the subject using the first terminal 2 shown in the second image with the gaze of a subject using the second terminal 3 that is different from the subject.

[Functional configuration of information processing device 1]
Next, a detailed description will be given of the configuration of the information processing device 1. Fig. 2 is a diagram showing the functional configuration of the information processing device 1. The information processing device 1 has a communication unit 11, a storage unit 12, and a control unit 13.
The communication unit 11 is a communication interface for transmitting and receiving data to and from the first terminal 2 and the second terminal 3 via a communication network such as the Internet or a mobile phone communication network.

The memory unit 12 is a storage medium that stores various data, and includes a ROM (Read Only Memory), a RAM (Random Access Memory), a hard disk, and an SSD (Solid State Drive). The memory unit 12 stores programs executed by the control unit 13. The memory unit 12 stores programs that cause the control unit 13 to function as an acquisition unit 131, an angle identification unit 132, an estimation unit 133, a color identification unit 134, a iris analysis unit 135, a size identification unit 136, a width identification unit 137, a correction unit 138, and an output unit 139.

The control unit 13 is, for example, a CPU (Central Processing Unit). The control unit 13 executes a program stored in the memory unit 12, thereby functioning as an acquisition unit 131, an angle identification unit 132, an estimation unit 133, a color identification unit 134, a iris analysis unit 135, a size identification unit 136, a width identification unit 137, a correction unit 138, and an output unit 139. Details of the acquisition unit 131, the angle identification unit 132, the estimation unit 133, the color identification unit 134, the iris analysis unit 135, the size identification unit 136, the width identification unit 137, the correction unit 138, and the output unit 139 are described below.

[Acquisition of a captured image of a subject]
The acquisition unit 131 acquires a plurality of images of the subject's face captured by an imaging device. Specifically, the acquisition unit 131 acquires in real time a plurality of images of the subject's face using the first terminal 2 captured at different times by an imaging device provided in the first terminal 2.

[Subject Eye Analysis]
Next, a process for analyzing the subject's eyes will be described. In correcting the subject's eyes, the angle identification unit 132, the estimation unit 133, the color identification unit 134, and the iris analysis unit 135 of the control unit 13 work together to analyze the subject's eyes shown in the first image acquired by the acquisition unit 131. A specific process for analyzing the subject's eyes will be described below.

The angle identification unit 132 analyzes each of the multiple first images acquired by the acquisition unit 131 to identify a lateral deviation angle, which is an angle at which the subject's face reflected in the first image is shifted horizontally when the front direction of the subject's face is used as a reference. The angle identification unit 132 analyzes each of the multiple first images using, for example, an existing face direction estimation technology to identify the lateral deviation angle of the subject's face reflected in each of the multiple first images. The angle identification unit 132 identifies a lateral deviation angle when the subject's face is facing more to the right than the front as a positive lateral deviation angle, and identifies a lateral deviation angle when the subject's face is facing more to the left than the front as a negative lateral deviation angle.

The estimation unit 133 estimates the ratio between the width of the pupil and the width of the white of the eye when the subject faces forward by analyzing the first image acquired by the acquisition unit 131. The estimation unit 133 estimates the ratios corresponding to each of the subject's left and right eyes by analyzing the first image acquired by the acquisition unit 131.

Specifically, first, the estimation unit 133 analyzes each of the multiple first images acquired by the acquisition unit 131 when the angle identification unit 132 identifies different lateral deviation angles, thereby identifying the image area of the subject's eyes that appears in each of the multiple first images. The estimation unit 133 analyzes the image area of the subject's eyes that appears in each of the multiple first images, thereby identifying the ratio of the width of the subject's pupil to the width of the white of the eye that appears in each of the multiple first images.

For example, the estimation unit 133 analyzes each of the images when the lateral deviation angle identified by the angle identification unit 132 is greater than or equal to -30° and less than -20°, greater than or equal to -20° and less than -10°, greater than or equal to -10° and less than 0°, greater than or equal to 0° and less than 10°, greater than or equal to 10° and less than 120°, and greater than or equal to 20° and less than 30°, thereby identifying the ratio between the width of the subject's iris and the width of the white of the eye shown in each of these multiple images.

The estimation unit 133 then averages the ratios of the width of the pupil to the width of the white of the eye determined in each of the multiple first images to estimate the ratio of the width of the pupil to the width of the white of the eye when the subject faces forward. Note that the estimation unit 133 may determine the ratio of the width of the pupil to the width of the white of the eye of the subject in the first image on the condition that the white of the eye is determined on both sides of the pupil of the subject in the image area of the subject's eyes in the first image. In this way, the information processing device 1 can accurately determine the width of the pupil in the image area of the subject's eyes. This allows the information processing device 1 to accurately determine the width of the white of the eye based on the image area of the eye and the determined width of the pupil, thereby improving the accuracy of determining the ratio of the width of the pupil to the width of the white of the eye.

The color identification unit 134 analyzes the first images acquired by the acquisition unit 131 to identify the color of the iris, the color of the left white of the eye, and the color of the right white of the eye of the subject. For example, the color identification unit 134 analyzes each of the multiple first images acquired by the acquisition unit 131 within a specified time period, and identifies the color of the iris, the color of the left white of the eye, and the color of the right white of the eye of the subject that appear in each of the multiple first images. Then, the color identification unit 134 identifies the average value of the iris colors identified from each of the multiple first images as the color of the iris of the left eye of the subject.

Similarly, the color identification unit 134 identifies the average value of the color of the left white of the left eye identified from each of the multiple first images as the color of the left white of the left eye of the subject, and identifies the average value of the color of the right white of the left eye identified from each of the multiple first images as the color of the right white of the left eye of the subject. The color identification unit 134 performs a similar process on the subject's right eye, and identifies the color of the pupil, the color of the left white of the eye, and the color of the right white of the subject's right eye.

The iris analysis unit 135 analyzes the first image acquired by the acquisition unit 131 to determine the ratio between the width of the iris of the subject and the width of the pupil in the iris. For example, the iris analysis unit 135 analyzes the first image acquired by the acquisition unit 131 within a specified time, and determines the maximum width of the image area corresponding to the iris of the subject's left eye shown in the first image, and the maximum width of the image area of the pupil included in the iris. The iris analysis unit 135 determines the ratio between the width of the iris of the subject's left eye and the width of the pupil based on the width of the image area corresponding to the identified iris and the width of the image area of the pupil. The iris analysis unit 135 performs a similar process on the subject's right eye, and determines the ratio between the width of the iris of the subject's right eye and the width of the pupil.

The iris analysis unit 135 analyzes the first image acquired by the acquisition unit 131 to identify the magnitude of light reflected in the iris and the position of the light in the iris. For example, the iris analysis unit 135 analyzes each of the multiple first images acquired by the acquisition unit 131 within a specified time, and identifies the position of the area in which the pixel value exceeds a specified threshold and the size of the area among the image areas corresponding to the iris of the subject's left eye shown in each of the multiple first images. The iris analysis unit 135 identifies the position of the light in the iris by, for example, identifying the relative position of the area corresponding to the light reflected in the iris with respect to the center position of the iris. The iris analysis unit 135 may identify an area whose size exceeds a specified size and whose pixel value exceeds a specified threshold as an area of light reflected in the iris. The iris analysis unit 135 performs a similar process on the subject's right eye to identify the magnitude of light reflected in the iris of the subject's right eye and the position of the light in the iris.

[Subject Eye Correction]
Next, a process for correcting the subject's eyes that appear in the image acquired by the acquisition unit 131 will be described. The angle specification unit 132, size specification unit 136, width specification unit 137, and correction unit 138 of the control unit 13 cooperate to correct the subject's eyes that appear in the second image acquired by the acquisition unit 131 after the analysis of the iris is completed. A specific process for correcting the subject's eyes will be described below.

The angle identification unit 132 analyzes the second image acquired by the acquisition unit 131 after the analysis of the iris is completed, thereby identifying a horizontal deviation angle, which is the angle at which the face reflected in the second image is shifted horizontally when the front direction of the subject's face is used as a reference. The angle identification unit 132 also analyzes the second image, thereby identifying a vertical deviation angle, which is the angle at which the face reflected in the second image is shifted vertically when the front direction of the subject's face is used as a reference.

The size determination unit 136 determines the size of the subject's eyeball and pupil by acquiring size information indicating the size of the subject's eyeball and pupil. It is known that there is little individual variation in the size of a person's eyeball and pupil. For this reason, size information indicating the size of an average person's eyeball and pupil is stored in the storage unit 12, and the size determination unit 136 determines the size of the average person's eyeball and pupil indicated by the size information stored in the storage unit 12 as the size of the subject's eyeball and pupil. Here, the size of the eyeball indicates the radius of the eyeball, and the size of the pupil indicates the maximum width of the pupil.

Based on the identified eyeball size, pupil size, and lateral deviation angle, and the estimated ratio, the width determination unit 137 determines the ratio between the pupil width, which is the width of the pupil of the subject when the subject looks straight ahead in the second image with the subject's face shifted by the lateral deviation angle with respect to the front direction, the left white width, which is the width of the left white of the eye reflected to the left of the pupil, and the right white width, which is the width of the right white of the eye reflected to the right of the pupil.

Below, the process by which the width determination unit 137 determines the width of the iris, the width of the left white of the eye, and the width of the right white of the eye will be explained with reference to the figures. FIG. 3 is a diagram explaining the state of the eyeballs when the subject's face is shifted to the side. FIG. 3 shows the state of the eyeballs when the subject's head H is viewed from above. It can be seen that FIG. 3 shows the left eye LE and the right eye RE. Also, in FIG. 3, the lateral shift angle at which the subject's face is shifted from the forward direction is z.

Figure 4 is an enlarged view of the subject's eyeball shown in Figure 3. In Figure 4, O indicates the center of the eyeball, and r indicates the radius of the eyeball. Figure 4 also shows the iris width, left white of the eye width, and right white of the eye width that are visible from the front direction, i.e., the iris width, left white of the eye width, and right white of the eye width when the subject's eye is reflected in the second image. Note that in the following explanation, an example of determining the ratio of the iris width, left white of the eye width, and right white of the eye width in the eyeball corresponding to the left eye will be explained, but the same process will be performed on the right eye to determine the ratio of the iris width, left white of the eye width, and right white of the eye width in the eyeball corresponding to the right eye.

Figure 5 is an enlarged view of the sector of the eyeball shown in Figure 4 that connects the center of the eyeball, the left end of the left white of the eye width, and the right end of the right white of the eye width. WL in Figure 5 indicates the left white of the eye width in real space, and WR indicates the right white of the eye width in real space. x indicates half the angle between the line connecting the left end of the iris width to the center O of the eyeball and the line connecting the right end of the iris width to the center O of the eyeball. y indicates half the angle between the line connecting the left end of the left white of the eye width to the center O of the eyeball and the line connecting the right end of the right white of the eye width to the center O of the eyeball. In the following explanation, x will also be referred to as the first angle, and y will be referred to as the second angle.

α indicates half the angle between the line connecting the left end of the left white of the eye width to the center O of the eyeball and the line connecting the right end of the left white of the eye width to the center O of the eyeball. β indicates half the angle between the line connecting the left end of the right white of the eye width to the center O of the eyeball and the line connecting the right end of the right white of the eye width to the center O of the eyeball.

The iris width is calculated by the following formula (1).

As shown in FIG. 5, 2α+x=yz, and α=(yzx)/2, so the left white of the eye width WL in real space is calculated by the following formula (2).

As shown in FIG. 5, 2β+x=y+z, and β=(y+z−x)/2, so the right white eye width WR in real space is calculated by the following formula (3).

6 is a diagram showing the relationship between the left white of the eye width in real space and the left white of the eye width in the second image, and the relationship between the right white of the eye width in real space and the right white of the eye width in the second image. Fig. 6(a) is a diagram showing the relationship between the left white of the eye width in real space and the left white of the eye width in the second image, and Fig. 6(b) is a diagram showing the relationship between the right white of the eye width in real space and the right white of the eye width in the second image. As shown in Fig. 6(a), the angle between the line connecting the center of the left white of the eye width WL and the center of the eyeball in real space and the line connecting the center of the iris width and the center of the eyeball is x + α. Therefore, the angle between the line extending the left white of the eye width WL in real space and the line extending the left white of the eye width is also x + α, and the left white of the eye width is expressed by the following formula (4).

As described above, since α=(yzx)/2, the left white width is expressed by equation (5) based on equations (2) and (4).

6B, the angle between the line connecting the center of WR to the center of the eyeball and the line connecting the center of the iris width to the center of the eyeball is x + β. Therefore, the angle between the line extending the right white of the eye width WR in real space and the line extending the right white of the eye width is also x + β, and the right white of the eye width is expressed by the following formula (6).

As described above, since β=(y+z−x)/2, the right white width is expressed by equation (7) based on equations (3) and (6).

First, the width determination unit 137 substitutes the eyeball size and the pupil size determined by the size determination unit 136 into formula (1) to calculate the first angle x. Note that if the eyeball size and the pupil size determined by the size determination unit 136 are fixed values, the first angle x is also a fixed value. Therefore, the first angle x may be stored in the storage unit 12, and the width determination unit 137 may acquire the first angle x stored in the storage unit 12.

The width determination unit 137 determines a second angle y based on the first angle x, the lateral deviation angle z, the ratio of the width of the iris to the width of the white of the eye estimated by the estimation unit 133, and equations (1), (5), and (7). If the ratio of the width of the white of the eye to the width of the iris is R, the following equation (8) holds based on the ratio of the width of the iris to the width of the white of the eye estimated by the estimation unit 133 and equations (1), (5), and (7).

The width determination unit 137 substitutes the specified first angle x, lateral deviation angle z, eyeball radius r, and ratio R of the white of the eye width to the iris width into equation (8) to calculate the second angle y. The width determination unit 137 substitutes the specified first angle x, second angle y, lateral deviation angle z, and eyeball radius r into equation (5) to calculate the left white of the eye width. Similarly, the width determination unit 137 substitutes the specified first angle x, second angle y, lateral deviation angle z, and eyeball radius r into equation (7) to calculate the right white of the eye width. The width determination unit 137 calculates the ratio of the iris width to the left white of the eye width to the right white of the eye width based on the calculated iris width, left white of the eye width, and right white of the eye width.

The correction unit 138 corrects the subject's eyes included in the second image acquired by the acquisition unit 131 so that the shapes of the pupil and white of the subject's eyes included in the second image are shapes that correspond to the ratio of the pupil width, left white width, and right white width specified by the width specification unit 137. For example, the correction unit 138 determines the display area of the left white of the left eye, the display area of the pupil, and the display area of the right white of the left eye based on the ratio of the pupil width, left white width, and right white width of the left eye specified by the width specification unit 137 in an image area showing the subject's left eye included in the second image.

The correction unit 138 corrects the subject's left eye by drawing the left white, black eye, and right white in the left white, black eye, and right white display areas determined for the left eye based on the color of the black eye, the color of the left white, and the color of the right white of the left eye identified by the color identification unit 134. The correction unit 138 also determines the left white, black eye, and right white display area for the subject's right eye, and draws the left white, black eye, and right white. In this way, the information processing device 1 can correct the subject's eyes so that the subject views the front.

When drawing the iris, the correction unit 138 draws the pupil at the center of the iris of the left eye based on the ratio between the width of the iris identified by the iris analysis unit 135 and the width of the pupil in the iris. The correction unit 138 also draws light for the iris drawn in the left eye based on the size and position of the light identified by the iris analysis unit 135. In this way, the information processing device 1 can correct the subject's eyes in accordance with how the subject's eyes are reflected in the first image.

Note that if the vertical deviation angle, which is the angle at which the subject's face in the second image is shifted up and down, is equal to or greater than a predetermined angle, correcting the subject's eyes may cause the subject's eyes to look unnatural. For this reason, the correction unit 138 may be configured to correct the subject's eyes in the second image when the magnitude of the vertical deviation angle identified by the angle identification unit 132 is equal to or less than a predetermined angle. In this way, the information processing device 1 can prevent the subject's eyes from being corrected to look unnatural.

[Image Output]
The output unit 139 outputs the image acquired by the acquisition unit 131 to the second terminal 3. Specifically, the output unit 139 outputs the first image acquired by the acquisition unit 131 as is to the second terminal 3. In addition, the output unit 139 outputs to the second terminal 3 a second image in which the subject's eyes have been corrected by the correction unit 138 after the subject's eyes have been analyzed based on the first image.

Figure 7 shows an example of correction of the subject's eyes. Figure 7(a) shows the second image before correction, and Figure 7(b) shows the second image after correction. It can be seen that the subject's line of sight in Figure 7(a) is not looking straight ahead, whereas in Figure 7(b) it has been corrected so that the subject's line of sight is looking straight ahead.

[Operation flow]
Next, a description will be given of the flow of processing in the information processing device 1. FIG 7 is a flowchart showing the flow of processing in the information processing device 1.

First, the acquisition unit 131 acquires a first image of the subject's face captured by an imaging device provided in the first terminal 2 (S1). The first image is used to analyze the subject's eyes.

Next, the estimation unit 133 estimates the ratio between the width of the iris and the width of the white of the eye when the subject faces forward by analyzing the first image acquired by the acquisition unit 131 (S2). Next, the color identification unit 134 analyzes the first image acquired by the acquisition unit 131 to identify the color of the subject's iris, the color of the left white of the eye, and the color of the right white of the eye (S3). Next, the iris analysis unit 135 analyzes the first image acquired by the acquisition unit 131 to identify the ratio between the width of the iris and the width of the pupil in the iris, and to identify the size and position of the light reflected in the iris (S4).

Next, the size determination unit 136 determines the size of the subject's eyeball and the size of the pupil by acquiring size information indicating the size of the subject's eyeball and the size of the pupil (S5). Note that the processes from S2 to S5 may be executed in parallel, or may be executed in an order different from that shown in FIG. 7.

Next, the acquisition unit 131 acquires a second image of the subject's face captured by an imaging device provided in the first terminal 2 (S6).
Next, the angle identifying unit 132 identifies the lateral deviation angle and the vertical deviation angle of the subject's face by analyzing the second image (S7).

The correction unit 138 determines whether the vertical deviation angle determined in S7 is less than a predetermined threshold value (S8). If the vertical deviation angle is less than the predetermined threshold value (YES in S8), the correction unit 138 proceeds to S9, and if the vertical deviation angle is equal to or greater than the predetermined threshold value (NO in S8), the correction unit 138 proceeds to S11.

In S9, the width determination unit 137 determines the ratio of the pupil width, left white of the eye width, and right white of the eye width when the subject looks straight ahead in the second image with the subject's face shifted by a lateral deviation angle based on the straight ahead direction, based on the determined eyeball size, pupil size, and lateral deviation angle, and the ratio of the estimated pupil width and white of the eye width.

Then, the correction unit 138 corrects the subject's eyes included in the second image acquired by the acquisition unit 131 (S10). Specifically, the correction unit 138 corrects the subject's eyes included in the second image so that the shapes of the pupil and the white of the eye in the second image correspond to the ratio of the pupil width, the left white of the eye width, and the right white of the eye width identified in S9. The correction unit 138 also draws the pupil, the left white of the eye, and the right white of the eye based on the color of the pupil, the color of the left white of the eye, and the color of the right white of the eye identified in S3. When drawing the pupil, the correction unit 138 draws the pupil in the center of the pupil of the left eye based on the ratio of the pupil width and the pupil width in the pupil identified in S4. The correction unit 138 also draws light for the pupil drawn in the left eye based on the size and position of the light identified in S4.

Next, the output unit 139 outputs the second image to the second terminal 3 (S11). If the subject's eyes have been corrected in S10, the output unit 139 outputs the second image in which the subject's eyes have been corrected to the second terminal 3, and if the subject's eyes have not been corrected, the output unit 139 outputs the second image acquired in S6 as is to the second terminal 3.

Then, the acquisition unit 131 determines whether or not a request to stop the process shown in FIG. 7 has been received (S12). If the acquisition unit 131 determines that a request to stop the process shown in FIG. 7 has been received (YES in S12), it ends the process related to this flowchart, and if it determines that a request to stop the process has not been received (NO in S12), it proceeds to S6.

[Effects of information processing device 1]
As described above, the information processing device 1 according to the present embodiment estimates the ratio of the width of the iris and the width of the white of the eye when the subject faces forward by analyzing the first image, and specifies the lateral shift angle, which is the angle at which the face reflected in the second image is shifted horizontally when the front direction of the face of the subject is used as a reference, by analyzing the second image acquired after acquiring the first image. The information processing device 1 specifies the ratio of the width of the iris, the width of the left white of the eye, and the width of the right white of the eye when the subject looks at the front direction in the second image in a state in which the face is shifted horizontally with the front direction as a reference based on the specified eyeball size, the size of the iris, and the lateral shift angle, and the estimated ratio, and corrects the second image so that the shape of the iris and the white of the eye of the subject included in the second image corresponds to the specified ratio of the width of the iris, the width of the left white of the eye, and the width of the right white of the eye. In this way, the information processing device 1 can align the gaze of the participant with the gaze of the other participants.

Furthermore, this invention will make it possible to contribute to Goal 9 of the United Nations' Sustainable Development Goals (SDGs), which is "Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote innovation and infrastructure."

Although the present invention has been described above using embodiments, the technical scope of the present invention is not limited to the scope described in the above embodiments, and various modifications and changes are possible within the scope of the gist of the invention. For example, all or part of the device can be configured by distributing or integrating functionally or physically in any unit. In addition, new embodiments resulting from any combination of multiple embodiments are also included in the embodiments of the present invention. The effect of the new embodiment resulting from the combination also has the effect of the original embodiment.

Reference Signs List 1 Information processing device 2 First terminal 3 Second terminal 11 Communication unit 12 Storage unit 13 Control unit 131 Acquisition unit 132 Angle identification unit 133 Estimation unit 134 Color identification unit 135 Iris analysis unit 136 Size identification unit 137 Width identification unit 138 Correction unit 139 Output unit

Claims (10)

an acquisition unit that acquires a plurality of images of a face of a subject captured by an imaging device;
a size determination unit that determines the size of the eyeball and the size of the iris by acquiring size information indicating the size of the eyeball and the size of the iris of the subject;
an estimation unit that estimates a ratio of a width of the black eye to a width of the white eye when the subject faces forward by analyzing the first image acquired by the acquisition unit;
an angle identification unit that identifies a lateral deviation angle, which is an angle at which the face shown in the second image is shifted in a lateral direction relative to a front direction of the face of the subject, by analyzing a second image acquired by the acquisition unit after acquiring the first image;
a width specifying unit that specifies a ratio between a width of the pupil of the subject when the subject views the front direction in the second image with the face deviated by the lateral deviation angle with respect to the front direction in the second image, a left white eye width indicating the width of the left white of the eye that is the white of the eye reflected to the left of the pupil, and a right white eye width indicating the width of the right white of the eye that is the white of the eye reflected to the right of the pupil, based on the specified eyeball size, the size of the pupil, and the lateral deviation angle and the estimated ratio;
a correction unit that corrects the second image so that the shapes of the iris and white of the subject's eyes included in the second image correspond to the specified ratio of the iris width, the left white of the eye width, and the right white of the eye width;
an output unit that outputs the second image in which the eye of the subject has been corrected by the correction unit;
An information processing device having the above configuration. The estimation unit estimates the ratios corresponding to the left and right eyes of the subject by analyzing the first image acquired by the acquisition unit.
The information processing device according to claim 1 . The estimation unit analyzes each of the multiple first images acquired by the acquisition unit to identify a ratio between the width of the iris and the width of the white of the eye of the subject shown in each of the multiple first images, and estimates the ratio between the width of the iris and the width of the white of the eye when the subject faces forward by averaging the ratios identified for each of the multiple first images.
The information processing device according to claim 1 . the angle identification unit identifies the lateral deviation angle of the face shown in the first image with respect to a front direction of the face of the subject by analyzing the first image;
The estimation unit estimates a ratio of a width of the iris to a width of the white of the eye of the subject captured in each of the first images by analyzing each of the first images when the angle identification unit identifies a different lateral deviation angle.
The information processing device according to claim 3 . The estimation unit estimates a ratio of a width of the iris and a width of the white of the eye of the subject shown in the first image, on the condition that the white of the eye is shown on both sides of the iris of the subject in the first image acquired by the acquisition unit.
5. The information processing device according to claim 3. a color specifying unit that specifies a color of the black eye, a color of the left white eye, and a color of the right white eye by analyzing the first image,
The correction unit corrects the color of the subject's eyes shown in the second image based on the color of the iris, the color of the left white of the eye, and the color of the right white of the eye identified by the color identification unit.
3. The information processing device according to claim 1 or 2. The method further includes a iris analysis unit that analyzes the first image to identify a size of light reflected in the iris and a position of the light in the iris,
The correction unit depicts the light with respect to the black eye based on the identified size and position of the light.
The information processing device according to claim 1 . Further comprising a iris analysis unit that identifies a ratio between a width of the iris and a width of a pupil in the iris by analyzing the first image,
The correction unit depicts the pupil relative to the iris based on a ratio between the identified width of the iris and a width of the pupil at the iris.
The information processing device according to claim 1 . the angle identification unit identifies a vertical deviation angle, which is an angle at which the face reflected in the second image is deviated in an up-down direction when a front direction of the face of the subject is used as a reference, by analyzing the second image;
The correction unit corrects the eyes of the subject appearing in the second image when the magnitude of the vertical deviation angle is equal to or smaller than a predetermined angle.
The information processing device according to claim 1 . The computer executes
acquiring a plurality of images of a face of a subject captured by an imaging device;
determining a size of the eyeball and a size of the iris of the subject by obtaining size information indicative of the size of the eyeball and the size of the iris of the subject;
estimating a ratio of a width of the iris to a width of the white of the eye when the subject is facing forward by analyzing the acquired first image;
A step of identifying a lateral deviation angle, which is an angle at which the face shown in the second image is shifted laterally with respect to a front direction of the face of the subject by analyzing a second image acquired after the first image is acquired;
a step of determining a ratio between a width of the subject's pupil, a left white eye width indicating the width of the left white eye, which is the white of the eye reflected to the left of the pupil, and a right white eye width indicating the width of the right white eye, which is the white of the eye reflected to the right of the pupil, when the subject views the front direction in the second image with the face shifted by the lateral shift angle with respect to the front direction; and
correcting the second image so that the shape of the iris and white of the subject's eye included in the second image corresponds to the specified ratio of the iris width, the left white of the eye width, and the right white of the eye width;
outputting the second image with the subject's eyes corrected;
An information processing method comprising the steps of:

Images