JP6793302B2 - 3D measuring device - Google Patents

Description

The present invention relates to a three-dimensional measuring device.

For the purpose of authentication, there are an increasing number of applications in which the three-dimensional shape of a face is measured and used in place of or in addition to the two-dimensional image of the face. In addition, a system for measuring and using the three-dimensional shape of a face has been developed for the medical and beauty fields, or for the production and adjustment of devices to be worn on the face such as eyeglasses. In this way, the three-dimensional shape measurement of the face is being applied to various fields.

On the face, not only parts that are almost in front of the face such as eyes, nose, and mouth, but also parts that are almost on the side of the face like ears and parts that are on the underside of the face like the underside of the chin. Is also included. In order to obtain a wide range of three-dimensional shapes of the face including parts other than the front, three-dimensional measurement of the face is performed from multiple different directions using a plurality of three-dimensional shape measuring devices, and the third order from these multiple directions is performed. The original measurement results are synthesized (for example, Patent Document 1).

In the method of synthesizing the three-dimensional measurement results from a plurality of different directions, the face to be measured needs to be located in a range where the fields of view of the three-dimensional shape measuring device in charge of each direction overlap. In order to satisfy this requirement, in the system of Patent Document 1, by providing a seat on which the person to be measured sits at a fixed position with respect to the two left and right three-dimensional shape measuring devices, the face of the person to be measured is a device of these two devices. Guarantee that the fields of view are located in the overlapping range.

Further, Patent Document 2 discloses a mechanism for giving guidance so that a person's face is in an appropriate position and orientation with respect to a camera in a system for appropriately taking a portrait photograph of a person. In this mechanism, the face of the subject is imaged with a stereo camera or the like, the distance of the face from the camera, the size on the screen, the bias of the position on the screen, etc. are obtained, and guidance information is generated from such information. ..

Japanese Unexamined Patent Publication No. 2005-106491 JP-A-2003-043263

The method of positioning the face by having the subject person sit on a seat fixed to the device increases the scale of the system. For example, when the three-dimensional measuring device is implemented as a small device, for example, a portable device, it is not appropriate to use such a method for face positioning.

The present invention provides a device that assists a person in easily aligning his or her face with an appropriate measurement position without using an external mechanism such as a seat that constrains or regulates the position of the person's face. The purpose is to provide.

The three-dimensional measuring device provided in one aspect of the present invention is different from a floodlight that projects structured light for three-dimensional measurement on the face and the face for the three-dimensional measurement. A plurality of cameras for three-dimensional measurement that capture images from a position, and a face to which the structured light is projected located at a predetermined measurement position for the three-dimensional measurement is used for the plurality of three-dimensional measurements. A three-dimensional measuring device that obtains the three-dimensional shape of the face from the image captured by the camera, and images of at least two alignment cameras that capture the face from different positions and at least two alignment cameras. It is a superimposing means for generating a superposed image by superimposing the obtained images so that a plurality of reference positions in each of the images overlap each other, and in each of the plurality of reference positions, the face is measured three-dimensionally. The superimposition means and the superimposition means at which each image of the plurality of reference parts of the face exists on the image captured by the at least two alignment cameras when the position is located at a predetermined measurement position. Includes a display means for displaying a combined image.

In the present invention, the display means may display the superimposed image by superimposing a guide frame indicating a range of appropriate positions of the face in the vertical and horizontal directions.

Further, the three-dimensional measuring device shifts two images captured by at least two of the two positioning cameras in a state of being aligned so that a plurality of reference positions in the two images overlap each other. When the amount is set to 0, one of the two images is shifted with respect to the other image, and the amount of shift when the images of a plurality of reference parts of the face on the two images overlap each other. It may further include a determination means for calculating τc and determining that the face is located at a predetermined measurement position when the obtained shift amount τc is equal to or less than a predetermined threshold value.

Further, the three-dimensional measuring device according to another aspect of the present invention is different from a floodlight that projects structured light for three-dimensional measurement on the face and the face for the three-dimensional measurement. A plurality of cameras for three-dimensional measurement that capture images from a position, and a face to which the structured light is projected located at a predetermined measurement position for the three-dimensional measurement is used for the plurality of three-dimensional measurements. It is a three-dimensional measuring device that obtains the three-dimensional shape of the face from the image captured by the camera, and is captured by two alignment cameras that capture the face from different positions and the two alignment cameras. When the amount of shift in a state where two images are aligned so that a plurality of reference positions in the two images overlap each other is set to 0, one image of the two images is set with respect to the other image. staggered, the shift amount τc when the image between the reference part of the multiple of the two of the face on the image overlap is calculated, if the calculated shift amount τc is equal to or less than a predetermined threshold value, the face Includes a determination means for determining that is located at the predetermined measurement position.

In addition, the three-dimensional measuring device brings the face closer to or further away from the three-dimensional measuring device in order to align the face at a predetermined measurement position depending on whether the shift amount τc is a positive value or a negative value. It may further include a guiding means for guiding the user.

Further, the determination means shifts one image with respect to the other image and calculates the correlation between the images or between the specific parts including the images of a plurality of reference parts in the images, and the correlation is the maximum. When this happens, it may be considered that the images of the plurality of reference parts on the two images overlap each other.

Further, the three-dimensional measuring device is a display means for displaying an superimposed image in which two images captured by two alignment cameras are superimposed so that a plurality of reference positions in the two images overlap each other. , A display means that superimposes a guide frame indicating the range of appropriate positions of the face in the vertical and horizontal directions on the superposed image, and a guide for the image of the face in the superposed image displayed on the display means. The first guidance that encourages the person to move the face to a position that fits in the frame is given to the person, and after receiving the start instruction from the person after the first guidance, the guidance for aligning the face to the predetermined measurement position. May further include a control means, which causes the guide means to execute.

According to the present invention, it assists a person to easily align his / her face position with respect to an appropriate measurement position without using an external mechanism such as a seat that restrains or regulates the position of the person's face. be able to.

It is a perspective view of the appearance of the three-dimensional face measuring apparatus of embodiment. It is a front view of the appearance of the three-dimensional face measuring apparatus of embodiment. It is a figure which shows typically the overlap of the field of view of the front, left side, and right side measurement units. It is a figure which shows the relationship between the position of the face of the subject person, and the position of the eye image of the subject person on the image plane of the front left camera. It is a figure which shows the relationship between the position of the face of the subject person, and the position of the eye image of the subject person on the image plane of the front left camera and the front right camera. It is a figure which shows typically the appropriate position of the eye in the image of the front left camera and the front right camera. It is a figure which shows typically the guide image which superposed the images of the front left camera and the front right camera so that the appropriate positions of eyes coincide with each other. In the guide image, it is a figure which shows typically the state when the positions of the eye images derived from the left and right cameras are substantially the same. It is a figure which shows typically another guide image which was superposed by inverting one of the images of a front left camera and a front right camera so that appropriate positions of eyes coincide with each other. It is a functional block diagram of the 3D face measuring apparatus of an embodiment. It is a figure which shows an example of the procedure of the process executed by the 3D face measuring apparatus of embodiment. It is a figure which shows the example of the image which combined the guide frame which becomes the reference of the alignment of a face in the vertical and horizontal directions with a guide image. It is a figure which illustrates the luminance distribution of the height near both eyes of the image taken by the front right camera. It is a figure which illustrates the luminance distribution of the height near both eyes of the image taken by the front left camera. It is a figure which illustrates the state where the correlation of the luminance distribution of the image of the front left camera and the front right camera is the largest. It is a functional block diagram of the 3D face measuring apparatus of the modification 1.

FIG. 1 shows a perspective view of the appearance of the three-dimensional face measuring device of the present embodiment, and FIG. 2 shows a front view.

This three-dimensional face measuring device (hereinafter, also simply referred to as "device") has a flat plate shape, and for example, its size is about 40 to 100 cm in width, about 30 to 70 cm in length, and about 3 to 7 cm in thickness (this numerical value is about 3 to 7 cm). Just an example). This device is portable and is installed at the place of use in a manner such as wall hanging.

A display unit 102 is provided in the center of the upper part of the front surface of the device, and illumination units 104, 106, and 108 are provided on the left, right, and below the display unit 102. The illumination units 104 to 108 project illumination light used when capturing a texture image of the face of the subject.

Further, in the vicinity of the periphery of the display unit 102, three openings 112 to 116 for the front three-dimensional measurement unit are provided.

Larger openings 126 and 136 on the left and right sides of the lower center illumination unit 108 cast structured light for three-dimensional measurement (for example, spatial coding pattern light or light cutting slit light). An opening provided in the skin on the front side of the device for the floodlight. Floodlights are provided inside the device at the back of the openings 126 and 136, respectively. In the present embodiment, the subject positions his / her face while viewing the display unit 102 from the front, and measures the positioned face. The floodlights project structured light through the left and right openings 126 and 136 from both the left and right sides of the face to be measured and in a direction in which the face is looked up from slightly below to above. As a result, the blind spot where the structured light is not projected is reduced, and the structured light is projected on the chin and its lower side, which is one of the features of the face shape.

The left and right openings 112 and 114 of the display unit 102 and the lower openings 116 are openings for a camera for three-dimensional measurement. A camera equipped with an image sensor or the like is provided inside the device behind each of the openings 112 to 116. The left and right openings 112 and 114 are located symmetrically with respect to the display unit 102, and the lower opening 116 is located on the symmetrical axis thereof. These cameras capture the illumination light of the illumination units 104 to 108 and the face on which the structured light is projected from the floodlight through the openings 126 and 136, respectively, through the corresponding openings 112, 114, 116. .. The openings 112 to 116, 126, 136 are covered with a transparent protective plate to prevent dust and the like from entering the inside of the device (this point also applies to the openings 122, 124, 132, and 134 described later). ).

Three-dimensional measurement is performed by a combination of a camera corresponding to the openings 112 and 116 and a structured light floodlight corresponding to the openings 126 and 136. Similarly, three-dimensional measurement is performed by a combination of a camera corresponding to the openings 114 and 116 and a structured light floodlight corresponding to the openings 126 and 136. From these, the three-dimensional shape when the face is viewed from the direction slightly deviated from the center of the face to the left and right can be obtained. Further, three-dimensional measurement may be performed by a set of two cameras corresponding to the openings 112 and 114. The cameras provided in the openings 112, 114, and 116 constitute a measurement unit that three-dimensionally measures the face in front of the display unit 102 from substantially the front direction. This measurement unit is called a front measurement unit.

The openings 122 and 124 are provided on both sides of the opening 126 having the floodlight under the illumination unit 106 on the left side of the display unit 102. Inside the device behind the openings 122 and 124, cameras equipped with an image sensor and the like are provided, respectively. These cameras image the face on which structured light is projected from the floodlight through the openings 126 and 136, respectively, through the openings 122 and 124, respectively. The two cameras corresponding to the openings 122 and 124 constitute a measurement unit that three-dimensionally measures the face in front of the display unit 102 from the lower left toward the center of the face. This measurement unit is called a left side measurement unit. This measuring unit measures the shape of the left side of the subject's face, including the shape of the lower left side of the chin.

Similarly, two openings 132 and 134 are provided on both sides of the opening 136 with the floodlight on the lower side of the illumination unit 104 on the right side of the display unit 102. Cameras are provided behind the openings 132 and 134, respectively. The two cameras corresponding to the openings 132 and 134 constitute a measurement unit that three-dimensionally measures the face in front of the display unit 102 from the lower right toward the center of the face. This measurement unit is called a right side measurement unit. This measuring unit measures the shape of the right side of the subject's face, including the shape of the lower right side of the chin.

In this example, the left side measurement unit and the right side measurement unit are arranged symmetrically with respect to the display unit 102, and their respective fields of view are also symmetrically arranged. This device includes at least two 3D shape data (including one corresponding to a pair of cameras with openings 112 and 116 and one corresponding to a pair of cameras with openings 114 and 116) obtained by the front measurement unit. , The three-dimensional shape data of the face of the subject is constructed by combining the three-dimensional shape data measured by the measurement units on the left and right sides, respectively.

In order to properly measure the three-dimensional shape of the subject's face with this device, it is necessary to align the face at an appropriate position where the fields of view of the front, left, and right measurement units overlap. As schematically shown in FIG. 3, the field of view 150 of the camera CM of the front measurement unit (the average field of view of the three cameras of the openings 112 to 116) and the field of view 154 of the camera CL of the left measurement unit (opening 122, The field of view 152 of the camera CR of the right side measurement unit (the average field of view of the two cameras of openings 132 and 134) is a certain distance range in front of the display unit 102, respectively. It overlaps with. If the face of the subject is in the face position A within the range where the fields of view 150 to 154 of the three measurement units overlap, the face can be appropriately measured in three-dimensional shape by all three measurement units. On the other hand, if the subject's face is at the face position X on the side closer to the device or the face position Y on the far side than the range where the fields of view 150 to 154 of the three measurement units overlap, the part of the face is three. Since it is out of the field of view of one or more of the measurement units, it becomes impossible to obtain an appropriate three-dimensional measurement result.

Therefore, in the present embodiment, the subject person himself / herself moves his / her face while looking at the image displayed on the display unit 102 without using a fixture such as a seat for fixing the position of the subject's face, which is appropriate. Align the face with the face position A. To support this alignment, the present embodiment uses images captured by two cameras corresponding to the left and right openings 112 and 114 of the display unit 102.

The principle of alignment support of this embodiment will be described.

First, consider the case of only one camera corresponding to the opening 112 on the left side. FIG. 4 schematically shows how the EyeL and EyeR of the face of the person who is the subject are projected onto the points PL-L and PL-R on the image plane Image Plane of the camera L on the left side. .. These points PL-L and PL-R are projection positions of both eyes EyeL and EyeR on the image plane when the face of the subject is at an appropriate face position A with respect to the measurement system including the camera L. If the focal length Focus and the image center of the camera L are known in advance and the coordinates of both eyes EyeL and EyeR at the appropriate face position A in the three-dimensional space are defined, both eyes EyeL at the appropriate face position A, The coordinates of the points PL-L and PL-R on which EyeR is projected on the image plane can be calculated. Hereinafter, these points PL-L and PL-R will be referred to as appropriate positions of the eyes on the image plane (of the camera L).

However, even if the points on the image plane of the camera L corresponding to both eyes EyeL and EyeR are at the appropriate positions PL-L and PL-R, the subject's face is at the appropriate face position A by itself. Is not always. Since the distance between Eye L and Eye R of both eyes of the subject is usually unknown, it is determined whether the current face position is an appropriate face position A or an inappropriate position such as face position B. I can't tell from the image of.

Next, with reference to FIG. 5, consider the case where there are two cameras, a camera L corresponding to the left opening 112 and a camera R corresponding to the right opening 114. For the camera R on the right side, the appropriate positions PR-L and PR-R of the eyes on the image plane when the subject's face is in the appropriate face position A can be calculated. As shown in FIG. 5, when the face is present at another face position other than the appropriate face position A, for example, the face position B or C, the projection points of the cameras L and R of both eyes EyeL and EyeR on the image plane are determined. It does not match one or more of the proper positions PL-L and PL-R of the eyes on the image plane of the camera L and the proper positions PR-L and PR-R of the eyes on the image plane of the camera R. Conversely, the projection points of both eyes EyeL and EyeR on the image planes of the cameras L and R are all of the appropriate eye positions PL-L, PL-R, PR-L and PR-R on those image planes. The match occurs only when the subject's face is in face position A.

From the above, the positions of both eyes EyeL and EyeR (four projection points) of the subject on the images taken by the cameras L and R are appropriate positions PL-L, PL-R, PR-L and PR-R. It can be seen that the face of the subject can be aligned with the appropriate face position A by adjusting the position of the face so as to match all of the above.

Therefore, in the present embodiment, both eyes EyeL and EyeR, which are relatively easy to recognize in the face, are used as reference parts for face alignment, and the positions of both eyes, which are the reference parts in the images taken by the cameras L and R, are set. It is possible to determine whether or not the face is in the appropriate position A by confirming that it substantially matches the appropriate positions PL-L, PL-R, PR-L and PR-R. The positions PL-L, PL-R, PR-L and PR-R on each image are the reference sites EyeL in the images of cameras L and R when the face is located at the appropriate face position A for measurement. And the position where the EyeR image is located, and these positions are also called the reference positions.

In one example, the images taken by the cameras L and R are displayed on the display unit 102, and while the subject person himself / herself confirms the image displayed on the display unit 102, it is as if he / she is looking in a mirror. Allows you to align your face naturally.

FIG. 6 is a schematic diagram of the appropriate eye positions PL-L, PL-R in the image of the camera L, and the appropriate eye positions PR-L, PR-R in the image of the camera R. For example, when the face moves to the right (Right) or left (Left), the image of the face in the images of the cameras R and L moves in the same direction as the direction of the movement. On the other hand, since the camera L and the camera R have different optical axis directions, the change direction of the face position on the image when the face is moved forward (Front) or rear (Back) is different between the camera L and the camera R. It will be in the opposite direction. For example, when the face moves forward, the face position in the image of the camera R moves to the right, and the face position in the image of the camera L moves to the left.

The "front-back" direction when moving the face referred to here is the front and rear directions as seen from the person of the subject who is looking at the display unit 102. From another point of view, the "front-back" direction is the front direction of the three-dimensional face measuring device (more strictly, the display unit 102), and the "front" direction or the "front" direction is displayed along the front direction. It is a direction approaching the unit 102, and the "rear" direction or the "rear" direction is a direction away from the display unit 102 along the front direction thereof. Further, the vertical direction and the horizontal direction are the vertical direction and the horizontal direction when viewed from the person of the subject who is viewing the display unit 102.

By the way, in this example, the image of the camera L and the image of the camera R are overlapped so that the appropriate eye positions PL-L and PR-L and PL-R and PR-R coincide with each other as shown in FIG. Together, the superimposed image (referred to as a guide image) is displayed on the display unit 102.

Here, the image coordinates of PL-L and PL-R in the superimposed image (guide image) are PL, and the image coordinates of PR-L and PR-R are PR. If the face is not in the proper face position A, the positions of both eyes do not match the PL-L, PL-R of the camera L and the PR-L, PR-R of the camera R. Therefore, in this case, even in the superimposed image, the positions of both eyes do not match PL and PR as shown in FIG. 7, and the eyes appear to be duplicated, resulting in a highly uncomfortable image. On the other hand, when the face is present at the appropriate face position A, the left eye and the right eye in the images of the cameras L and R overlap each other in PL and PR even on the superimposed image as shown in FIG. 8, so that the image with the least discomfort It becomes.

Summarizing the above, in this example, first, an image as shown in FIG. 7 in which the images taken by the two cameras L and R are superimposed as described above is displayed on the display unit 102 and presented to the subject person. Next, the subject person moves his / her face back and forth and left and right while looking at the presented image, and searches for the face position where both eyes overlap and the feeling of discomfort is minimized as shown in FIG. The face position determined in this way is the appropriate face position A.

In addition, in order to generate a superposed image with less discomfort, the original camera image may be superposed after being processed such as left-right inversion, rotation, and projective transformation.

Further, in this example, as shown in FIG. 7, when the face is moved in the front-rear direction, it moves in the opposite directions on the images of the camera L and the camera R, so that the change on the superimposed image is large, but the face. When is moved in the left-right direction, both images move in the same direction, so there is relatively little change on the superimposed image.

However, as shown in FIG. 9, when only one of the images of the camera L and the camera R is flipped horizontally, this relationship is also reversed, and the more the face is moved in the left-right direction, the larger the change on the superimposed image becomes. .. Therefore, depending on whether it is more important to match the front-back direction or the left-right direction of the face position, it may be decided whether or not one of the original camera images is horizontally inverted when the images are superimposed. Alternatively, if the superimposed images of both the case of left-right reversal and the case of no left-right inversion are presented at the same time, the front-back and left-right can be precisely matched. For example, the display area of the display unit 102 is divided into a first and a second area, and an image superimposed on the first area without left-right inversion is superimposed on the second area by inverting one side on the left-right Each of the images may be displayed.

FIG. 10 shows a functional block diagram of the three-dimensional face measuring device in this example. This illustrates the functional configuration of the device whose appearance is shown in FIGS. 1 and 2.

In FIG. 10, the front left camera 212 and the front right camera 214 are cameras in the openings 112 and 114 on the left and right sides of the display unit 102 shown in FIG. 1, respectively. Further, the front lower camera 216 is a camera in the opening 116 on the lower side of the display unit 102. The left side first camera 222 and the left side second camera 224 are cameras in the two openings 122 and 124 on both sides of the left side floodlight opening 126, and the right side first camera 232 and the right side second camera. Reference numeral 234 is a camera in two openings 132 and 134 on both sides of the right floodlight opening 136.

The guide image generation unit 300 generates a guide image to be displayed on the display unit 102 for face alignment from the images captured by the front left camera 212 and the front right camera 214 in the alignment mode. In the simplest example, the guide image is simply an image obtained by superimposing an image captured by the front left camera 212 and an image captured by the front right camera 214. The guide image may be generated by converting the images of the front left camera 212 and the front right camera 214 into an image when both are viewed from the front direction by a projective transformation or the like, and superimposing the images. In the alignment mode, the floodlights in the openings 126 and 136 do not project the structured light for three-dimensional measurement, and the illumination units 104 to 108 are projecting the illumination light, and the front left camera 212 and The front right camera 214 takes an image.

The three-dimensional face shape component 302 combines the images of the front measurement unit, the left side measurement unit, and the right side measurement unit in the measurement mode, and applies a three-dimensional measurement algorithm such as a spatial coding method to obtain three-dimensional shape data. To generate. In the measurement mode, structured light is projected onto the face of the subject from the floodlights in the openings 126 and 136, and the faces in this state are imaged by the cameras 212, 214, 216, 222, 224, 232, and 234. Then, the three-dimensional face shape component 302 generates one three-dimensional shape data from the set of images of the front left camera 212 and the front lower camera 216, and one from the set of images of the front right camera 214 and the front lower camera 216. Generate two 3D shape data. Further, the three-dimensional face shape component 302 generates one three-dimensional shape data from a set of images captured by the left first camera 222 and the left second camera 224, and further, the right first camera 232 and the right second camera 232. One three-dimensional shape data is generated from a set of images captured by 234. Then, the three-dimensional face shape constituent unit 302 has two three-dimensional shape data obtained from the images of the three front cameras, one three-dimensional shape data obtained from the pair of cameras on the left side, and a pair of cameras on the right side. By combining with one three-dimensional shape data obtained from the above, three-dimensional shape data of the face including the jaw, ears, etc. in addition to the front of the face is constructed. Since the relationship between the individual coordinate systems of each camera 212 to 216, 222 to 224, and 232 to 234 to the world coordinate system is known, the images in the individual coordinate systems of each camera are coordinate-converted to the same world coordinate system. By applying the three-dimensional system measurement algorithm, three-dimensional shape data in the same coordinate system can be obtained, and the three-dimensional shape data may be synthesized.

The measurement control unit 304 controls each functional element in the device in order to generate three-dimensional face shape data of the subject person. When measuring the face of the subject person, the measurement control unit 304 first executes the process of the alignment mode, and then executes the process of the measurement mode. In the alignment mode, the measurement control unit 304 controls the front left camera 212 and the front right camera 214 to take an image for a guide image, and combines the images captured by the two cameras 212 and 214 to form a guide image. Is generated and displayed on the display unit 102. Further, in the measurement mode, the measurement control unit 304 causes the floodlights provided in the openings 126 and 136 to project the structured light, and causes the cameras 212 to 216, 222 to 224, and 232 to 234 to perform imaging. The three-dimensional face shape component 302 is made to generate three-dimensional shape data from the images of each of these cameras.

Next, an example of the procedure for measuring the face shape by the three-dimensional face measuring device of the present embodiment will be described with reference to FIG.

This procedure starts when, for example, a person as a subject instructs the device to start measurement. This start instruction may be given, for example, by pressing the measurement start button provided on the device or instructing the start by voice (in the latter case, the device recognizes the measurement start instruction by voice). Therefore, the microphone and the instruction content are equipped with software for recognition).

When the process is started, the measurement control unit 304 first controls the alignment mode. That is, the measurement control unit 304 turns on the illumination units 104 to 108 (S10), and images the front left camera 212 and the front right camera 214 in the alignment mode (that is, the image capture result without projecting the structured light). Output video) is instructed. The guide image generation unit 300 generates the above-mentioned guide image from the images of the front left camera 212 and the front right camera 214, and displays the guide image on the display unit 102 (S12). The guide image is displayed on the display unit 102 as a moving image, for example. The subject person moves the position of the face while looking at the guide image displayed on the display unit 102, and finds the position on the guide image in which the blurring of both eyes of his / her face is the least and the image state with the least discomfort is obtained. A voice output mechanism may be provided in the device, and the measurement control unit 304 may output a guidance voice such as "Please move your face to a position where your eyes are not duplicated on the screen" when S12 is executed. Alternatively, the same guidance may be displayed as characters on the display unit 102. The measurement control unit 304 waits for the subject person to instruct the start of the three-dimensional measurement (S14).

When the alignment of the face is completed, the subject person inputs a start instruction of the three-dimensional measurement to the device by voice, pressing a button, or the like. Then, the determination result of S14 becomes Yes, and the measurement control unit 304 shifts to the measurement mode. In the measurement mode, the measurement control unit 304 causes the floodlights provided in the openings 126 and 136 to project the structured light, and causes the cameras 212 to 216, 222 to 224, and 232 to 234 to perform imaging (S16). ). Then, the three-dimensional face shape component 302 combines the images captured by these cameras for each pair such as a pair of the front left camera 212 and the front lower camera 216 and a pair of the front right camera 214 and the front lower camera 216. Apply a three-dimensional measurement algorithm. As a result, three-dimensional shape data for each pair is generated. The three-dimensional face shape component 302 completes the three-dimensional shape data of the face of the subject person by synthesizing the three-dimensional shape data for each pair (S18). In the three-dimensional measurement process, a facial texture image is acquired by lighting the lighting units 104 to 108 without projecting structured light and taking a picture, and the texture information obtained from this texture image is obtained. May be combined with the three-dimensional shape data.

As described above, in the present embodiment, the subject person himself / herself can see the display state of both eyes on the guide image displayed on the display unit 102, and can align his / her face with an appropriate face position. ..

In the above, two cameras, the front left camera 212 and the front right camera 214, were used to generate the guide image, but another pair of two cameras (for example, the first camera on the left side) was used to generate the guide image. A pair of 222 and the right first camera 232, etc.) may be used. Further, the guide image may be generated by using three or more cameras. When using three or more cameras, the guide image generator 300 sets the guide image generator 300 so that the appropriate eye positions PL-L and PR-L and PL-R and PR-R on the image plane of each camera overlap each other. A guide image in which the images of these cameras are superimposed is generated and displayed on the display unit 102. Further, in the above example, a guide image for alignment is generated using a camera used for three-dimensional measurement such as a front left camera 212 and a front right camera 214, but a camera dedicated to alignment is separately used. A guide image may be generated from the images taken by these dedicated cameras.

Further, in the above example, the appropriate positions of both eyes PL-L, PR-L, PL-R, and PR-R were used as a reference when superimposing images from a plurality of cameras to generate a guide image. This is just one example. Alternatively or in addition to this, a site other than the eyes, such as the contour of the nose or chin, may be used as the criterion for superposition. In this case, the superposition of the plurality of images at the time of generating the guide image is a method of matching the average positions of the plurality of reference parts in the images, and the images so that the appropriate positions of the plurality of reference parts match. May be transformed by a projective transformation or the like and then superimposed. When superimposing images, all of the plurality of reference points may not be treated equally, but weighting may be performed according to the importance. For example, the plurality of images are superposed so that the sum of the products of the misalignment of the same reference portions and the weights between the images is the smallest.

Further, in the above embodiment, the guide display for face alignment may be combined and displayed with the guide image displayed on the display unit 102.

For example, as shown in FIG. 12, the guide image generation unit 300 displays the guide frame 172 in the guide image 170. The guide frame 172 is a frame line surrounding a range slightly larger than the existing range of the superimposed face images in the guide image 170 when a face of a standard size is located at an appropriate face position A. The subject person adjusts the position of his / her face so that the superposed and double-copied face images in the guide image 170 displayed on the display unit 102 are approximately fitted in the guide frame 172. When the face image is substantially contained in the guide frame 172, the face of the subject is positioned at least in an appropriate range in the left-right and up-down directions. After this, the subject person moves the face approximately forward or backward toward the display unit 102, and if necessary, finely adjusts the position of the face in the vertical and horizontal directions to minimize discomfort in both eyes. Find the position.

As an example of the face alignment procedure, the subject person is made to adjust the face position so that the face image displayed on the display unit 102 fits in the guide frame 172 in the first step, and then the left and right are left and right in the second step. The face alignment in the front-rear direction may be executed by the guide image in which the images of the front left camera 212 and the front right camera 214 are superimposed. In this case, in the first stage, the image of the front lower camera 216 may be displayed on the display unit 102 instead of the guide image in which the images of the front left camera 212 and the front right camera 214 are superimposed. Then, when the first stage alignment is completed, the subject person notifies the three-dimensional face measuring device by voice or the like, and the device displays a guide image for the second stage in response to this notification. It may be displayed in the unit 102.

In this example, after positioning the face within a certain correct range in the left-right and up-down directions by positioning with reference to the guide frame 172, it is possible to focus on the alignment in the front-back direction, so that the face can be aligned more than without the guide frame 172. Alignment work becomes easier.

<Modification example 1>
In the above-described embodiment, the subject person moves his / her face while looking at the guide image to determine whether or not the person can be properly aligned. On the other hand, in this modified example, the three-dimensional face measuring device determines whether the position of the face of the subject is appropriate by calculation. In addition, when the position of the face deviates from the appropriate face position, the device guides which way to move the face.

The method described below is based on the premise that the position of the subject's face is already properly aligned in the left-right (horizontal) direction and the up-down direction. Under this premise, the device assists in aligning the face in the anteroposterior direction (direction approaching or moving away from the device), and determines whether or not the face is properly aligned. The alignment of the face in the left-right and up-down directions may be performed by, for example, the method performed by the person himself / herself adjusting the position of the face while looking at the guide frame 172 as in the above-mentioned example.

The method of this modification will be described below.

FIG. 13A shows an image taken by the front right camera 214 (Camera R). The graph shown in FIG. 13 (b) is a plot of the distribution of the luminance values at y coordinates and y = y _iR near both eyes (EyeR, EyeL) in this image. Since the human pupil is black, this graph shows a luminance distribution with reduced luminance in both eyes. Let this luminance distribution be f _R (x). Similarly, FIG. 14A shows an image taken by the front left camera 212 (Camera L), and FIG. 14B schematically shows the luminance distribution at y = y _iL . Let this luminance distribution be f _L (x). Figure 15 shows a luminance distribution f _R (x), the manner in which the luminance distribution f _L (x) superimposed luminance distribution f _L (x + τ) obtained by shifting tau in the x direction. In FIG. 15, the shift amount τ is determined so that the x-coordinates of Eye _L and Eye _R of the luminance distribution f _R (x) and the luminance distribution f _L (x + τ), respectively, match. At this time, since the images of the camera L and the camera R are in a state where both eyes are in agreement as shown in FIG. 8, that is, a state in which the face is located at an appropriate face position A, the value of τ in this state is theoretical. It can be obtained by calculation or actual measurement. Therefore, τ in this state is defined as a reference value, for example, 0.

Next, the correlation between the brightness distribution f _R (x) and f _L (x + τ) with respect to the image of the actual subject.
Is calculated, and the value τc of the shift amount τ when this correlation is maximized is obtained. Here, the shift amount τ is the shift amount when the brightness distribution f _L (x) of the camera L on the left side is shifted in the x direction.

At this time, the luminance distributions f _R (x) and f _L (x + τ) are considered to be in a state where the positions of Eye R and Eye _L are the same as shown in FIG. Note that f _L (x + τ) is a luminance distribution when the image of the camera L in FIG. 6 is translated in the left-right direction by an amount of τ. When the face is located at an appropriate face position A, the shift amount τc at the maximum correlation is 0.

Summarizing the above, in this modified example, first, τc, which is the amount of shift when the correlation value of the equation (1) between the image of the camera R and the image of the camera L is maximized, is calculated. As shown in the image of the camera L in FIG. 6, if τc is a positive value with respect to the reference value, the position of the face is moved backward, and if τc is a negative value, the subject person is guided to move the face forward.

When the face appropriately positioned in the vertical and horizontal directions with reference to the guide frame 172 moves in the front-rear direction, the images of the eyes (and further the face) are mutual in the image of the camera L and the image of the camera R. Move in the opposite direction. Further, it is known in which direction the image of the eye moves in each of the images of the cameras L and R depending on whether the face moves forward or backward. Further, when the face of the subject is at an appropriate face position A, the shift amount τc at the maximum correlation is a reference value, that is, for example, 0. From these facts, it is known whether the face is located in front of or behind the appropriate face position A when the shift amount τc at the maximum correlation is a positive value or a negative value with respect to the reference value. It is the opposite for positive and negative values. Therefore, if the positive and negative signs of the shift amount τc at the maximum correlation are known, it can be determined whether the face should be moved back or forth from the sign. In the example of FIG. 6, when the face properly positioned vertically and horizontally moves forward, the face in the image of the camera L moves to the left (negative direction of the x-axis), and the face in the image of the camera R moves to the left. Is an example when the relationship of moving to the right holds.

After that, images are acquired from the camera R and the camera L again, the amount of shift τc that maximizes the correlation value of the equation (1) is obtained, and the same guidance as above for the back-and-forth movement of the face is performed according to the sign of the τc. By repeating this, if the shift amount τc at the maximum correlation can be regarded as substantially 0 (more specifically, it is equal to or less than a predetermined threshold value close to 0), the face position is appropriate for the subject person. It is judged that the position is aligned with A, and the subject person is informed that it is not necessary to move the face any more.

FIG. 16 illustrates the functional configuration of the three-dimensional face measuring device for this modification. The same elements as those in the three-dimensional face measuring device of the above embodiment shown in FIG. 10 are designated by the same reference numerals, and the description thereof will be omitted.

The three-dimensional face measuring device shown in FIG. 16 has a face position guiding unit 310 instead of the guide image generating unit 300 in the device of FIG. When the subject person instructs the start of measurement, the measurement control unit 304 operates the front left camera 212 and the front right camera 214 in the alignment mode. The face position guidance unit 310 receives images from the front left camera 212 and the front right camera 214, respectively, and performs a correlation calculation between the luminance distributions at the heights near the eyes of both images as described above, and the correlation value becomes maximum. Calculate the shift amount τ. Then, it guides which direction to move the face back and forth according to the positive or negative of the value of τ with respect to the reference value. In this guidance, the degree of the amount of movement of the face back and forth (“slight”, “large”, etc.) may be indicated according to the magnitude of the absolute value of τ. The guidance method is not particularly limited. For example, the guidance may be performed by voice, or may be performed by displaying a guidance message on the display unit 102. Then, when the shift amount τ becomes a sufficiently small value (below the threshold value), the face position guidance unit 310 guides the face not to move because it starts measurement, and notifies the measurement control unit 304 of the completion of alignment. .. Upon receiving this notification, the measurement control unit 304 operates the floodlights in the openings 126 and 136 and the cameras 212 to 216, 222 to 224, and 232 to 234 in the measurement mode. The three-dimensional face shape component 302 generates three-dimensional shape data from the images captured by the cameras 212 to 216, 222 to 224, and 232 to 234, synthesizes the three-dimensional shape data, and finally forms the face shape. Generate data.

In the above description of the modified example, it is described that the correlation of the image (luminance distribution) near the eyes is used. Here, the vicinity of the eyes means the appropriate eye positions PL-L, PL-R, as shown in FIG. 5, assuming that the faces have been properly aligned in the horizontal and vertical directions. It may be specified by referring to the y-coordinates of PR-L and PR-R.

Further, the method of the modified example can be realized not only near the eyes but also by using parts such as the nose, mouth, and contour as reference parts. In this case, among the images of the front left camera 212 and the front right camera 214, the shift amount τc that maximizes the correlation between the portions including the reference portion may be obtained. Further, if the correlation is taken with respect to the entire face rather than the specific range of the face, it is not necessary to obtain the position of each part such as the eyes.

<Modification 2>
In the above-described modification, the correlation between the images of the front left camera 212 and the front right camera 214 was used to guide the moving direction of the face position and determine whether or not the face position was appropriate. On the other hand, in this modification 2, the feature parts (eyes, nose, mouth, etc.) in the face are used from the images of the front left camera 212 and the front right camera 214 by using a known image feature extraction technique. Find the coordinates of the contours, etc.) and find the difference in the coordinates of those feature parts between the two images. This difference can be used in the same manner as the shift amount τ in the modified example. The featured portion of the face may be extracted from the image by a known image processing technique, for example, feature point extraction by edge detection, template matching, deep learning, or the like.

Further, in the first and second modifications, the images of two cameras are used, but the same processing may be performed by using three or more cameras. That is, for each combination of two of the three or more cameras, the evaluation value corresponding to the shift amount τ of one image when the correlation between the images of those combinations is maximized is obtained by the above method. .. Depending on the combination of the number of feature points and the number of cameras, multiple evaluation values corresponding to τ can be obtained. For example, voting (using the most moving direction of the face) and the magnitude of the evaluation value (absolute value is the most). The movement direction may be appropriately determined by a method such as (adopting a movement direction based on a large evaluation value).

Further, the display of the guide image of the above embodiment may be combined with the guidance according to the modification 1 or 2. In this method, the face position guide unit 310 guides the subject person in parallel with aligning the face position while looking at the guide image generated by the guide image generation unit 300. The face position guidance unit 310 includes guidance on whether to move the face back and forth, and guidance indicating that the face has been positioned at an appropriate position (the shift amount τc is equal to or less than the threshold value). You only have to do at least one.

Further, the face position guide unit 310 responds to the magnitude (absolute value) of the shift amount τc (the shift amount when the correlation value is maximum or when the images of the reference portions recognized from the two images substantially overlap). You may guide how much you should move your face forward or backward. In this case, the face position guidance provides information on the correspondence between the magnitude of the shift amount τc and the information on the degree of movement of the face (for example, a numerical value such as several centimeters, or a word indicating the degree of "a little" or "large"). It may be set in the unit 310, and the face position guidance unit 310 may perform guidance according to the information.

102 Display, 104, 106, 108 Lighting, 112, 114, 116, 122, 124, 126, 132, 134, 136 Opening, 120, 130 Opening set, 170 guide image, 172 guide frame, 212 Front left Camera, 214 Front right camera, 216 Front lower camera, 222 Left left first camera, 224 Left second camera, 232 Right first camera, 234 Right second camera, 300 Guide image generator, 302 Three-dimensional face shape component, 304 Measurement control unit, 310 Face position guidance unit.

Claims (12)

A floodlight that projects structured light for three-dimensional measurement onto the face,
A plurality of cameras for three-dimensional measurement that capture the face from different positions for the three-dimensional measurement, and
The three-dimensional shape of the face is obtained from images taken by the plurality of cameras for three-dimensional measurement of the face to which the structured light is projected, which is located at a predetermined measurement position for the three-dimensional measurement. The 3D measuring device you are looking for
At least two alignment cameras that capture the face from different positions,
An overlay means for generating an superimposed image by superimposing images captured by at least two alignment cameras so that a plurality of reference positions in each of the images overlap each other, and each of the plurality of reference positions. Is a position where each image of a plurality of reference parts of the face exists on an image captured by the at least two alignment cameras when the face is positioned at a predetermined measurement position for the three-dimensional measurement. The superposition means and
A display means for displaying the superimposed image and
3D measuring device including. The three-dimensional aspect according to claim 1, wherein the display means superimposes and displays a guide frame indicating a range of appropriate positions of the face in the vertical and horizontal directions on the superimposed image. Measuring device. The three-dimensional measuring device according to claim 1 or 2, wherein the at least two alignment cameras are alignment cameras provided on the left side and the right side of the display means in the front direction, respectively. The three-dimensional measuring device is
The image captured by one of the positioning cameras provided on the left side and the right side with respect to the front direction of the display means and the image captured by the other camera are superimposed. Further provided with means for generating a combined second superimposed image,
The display means displays the superposed image and the second superposed image.
The three-dimensional measuring device according to claim 3, wherein the three-dimensional measuring device is characterized. When the amount of shift in a state where two images captured by two of the at least two alignment cameras are aligned so that a plurality of reference positions in the two images overlap each other is set to 0, the above-mentioned One of the two images was shifted with respect to the other image, and the amount of shift τc when the images of the plurality of reference parts of the face on the two images overlapped with each other was calculated and obtained. A determination means for determining that the face is located at the predetermined measurement position when the shift amount τc is equal to or less than a predetermined threshold value.
The three-dimensional measuring device according to any one of claims 1 to 4, further comprising. Depending on whether the shift amount τc is a positive value or a negative value, it guides whether the face is brought closer to or farther from the three-dimensional measuring device in order to align the face with the predetermined measurement position. The three-dimensional measuring device according to claim 5, further comprising a guiding means for The three-dimensional measuring device according to any one of claims 1 to 6, wherein at least two of the plurality of three-dimensional measuring cameras are also used as the at least two positioning cameras. A floodlight that projects structured light for three-dimensional measurement onto the face,
A plurality of cameras for three-dimensional measurement that capture the face from different positions for the three-dimensional measurement, and
The three-dimensional shape of the face is obtained from images taken by the plurality of cameras for three-dimensional measurement of the face to which the structured light is projected, which is located at a predetermined measurement position for the three-dimensional measurement. The 3D measuring device you are looking for
Two alignment cameras that capture the face from different positions,
When the amount of shift in the state where the two images captured by the two alignment cameras are aligned so that the plurality of reference positions in the two images overlap each other is set to 0, the two images are included in the two images. shifting one of the images relative to the other image, they shift amount τc when the image between the multiple reference position of the face on the two images are overlapped and calculated, the obtained shift amount τc is given When the value is equal to or less than the threshold value of, the determination means for determining that the face is located at the predetermined measurement position, and
3D measuring device including. The determination means shifts the one image with respect to the other image, calculates the correlation between the images, or calculates the correlation between the images and specific parts of the images including the images of the plurality of reference sites, and calculates the correlation. The three-dimensional measuring device according to claim 8, wherein the images of the plurality of reference parts on the two images are regarded as overlapping when the value is maximized. Depending on whether the shift amount τc is a positive value or a negative value, it guides whether the face is brought closer to or farther from the three-dimensional measuring device in order to align the face with the predetermined measurement position. The three-dimensional measuring device according to claim 8 or 9, further comprising a guiding means. It is a display means for displaying a superposed image in which two images captured by the two alignment cameras are superposed so that a plurality of reference positions in the two images overlap each other. A display means for overlaying and displaying a guide frame indicating a range of appropriate positions of the face in the vertical and horizontal directions.
A first guide is given to the person to encourage the person to move the face to a position where the image of the face in the superimposed image displayed on the display means fits in the guide frame, and after the first guide. A control means for causing the guidance means to perform guidance for aligning the face with the predetermined measurement position upon receiving a start instruction from the person.
The three-dimensional measuring device according to claim 10, further comprising. The three-dimensional measuring device according to any one of claims 8 to 11, wherein two of the plurality of three-dimensional measuring cameras are also used as the two positioning cameras.