JP5206516B2 - Imaging auxiliary device, program, and imaging system - Google Patents

Description

  The present invention relates to a photographing assistance device that assists photographing by notifying the photographing subject so that the photographing subject can easily move his / her face to a predetermined photographing position.

  2. Description of the Related Art Conventionally, a technique has been proposed in which a subject's face is photographed and changes in facial orientation and facial expression are captured from the photographed image. For example, a device is proposed that detects a subject's dozing state by capturing a facial image of the subject (a vehicle driver) and capturing changes in the subject's facial expression using a face shape model. (See Patent Document 1).

  Patent Document 1 describes a method of capturing the facial expression of a subject using an image recognition technique based on a face shape model. In the image recognition method using a face shape model, a target person who has a face shape close to the face shape model that the computer system performing image processing has in advance can capture facial expressions with high accuracy. For subjects with significantly different shapes, the estimation accuracy of changes in facial orientation and facial expression may be low.

  In view of this, a method has been proposed in which a face shape model using a front face image of a target person acquired in real time is used to deal with a face shape target person that the computer system does not have as a face shape model ( Non-patent document 1). At this time, by using the front face image photographed from the front of the subject with high accuracy, it is possible to improve the estimation accuracy of the change in the face direction and the facial expression.

  As a technique for photographing a front face image, a technique has been proposed in which it is determined that the driver is facing the front based on the turning angle of the steering wheel or the acceleration of the accelerator, and the image at that time is acquired as a front face image (patent) Reference 2)

Japanese Patent Application No. 2007-276511 JP 2008-197722 A

I. Matthews, T. Ishikawa and S. Baker: "The template update problem", Proceedings of the British Machine Vision Conference, September, 2003.

  In order to improve the estimation accuracy of facial expression changes using the face shape model, it is necessary to acquire a frontal face image taken from the front of the subject with high accuracy as described above. In order to accurately shoot from the front using the photographic device, it is convenient for the subject to actively move to a predetermined shooting position and face his face in a predetermined direction.

  However, since the technique of the above-mentioned Patent Document 2 is to shoot by assuming that the driver is facing the front when the driver has a high possibility of facing the front, On the other hand, since it is not required to move the face in a predetermined shooting position and face direction suitable for shooting, it is difficult to accurately capture a front face image.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide a photographing assistance device, a program, and a photographing system that allow a photographing subject to recognize a photographing position suitable for photographing. It is.

  According to another aspect of the present invention, there is provided a photographing assisting device according to a first aspect of the present invention, in which a photographing subject has a predetermined photographing position (a position and an orientation in a desired positional relationship with respect to the photographing device). This is a device for informing the subject to be photographed so that the face can be easily moved to the position of the face directly in front of the device and the orientation of the face. In this photographing assisting device, based on the photographed image indicating the face of the subject to be photographed acquired by the image acquiring means, the extracting means extracts a plurality of feature points on the face in the photographed image.

  Further, the distance measuring means measures the distance between the feature point extracted by the extracting means and a reference point predetermined on the captured image. The reference point here is a point that exists corresponding to each of the above-described feature points and has the smallest distance from the feature point when the face of the person to be photographed is at the above-described predetermined photographing position.

The notifying means is configured to output image information indicating an image obtained by superimposing the feature point and the reference point on a plane to a display device capable of displaying an image .
In the photographing auxiliary device configured as described above, when a photographed image of the face of the person to be photographed photographed by the photographing apparatus is acquired, if the face of the person to be photographed is at the predetermined photographing position described above, the photographed image The distance between the feature point on the face and the predetermined reference point becomes small. On the other hand, when the face of the person to be photographed is at a position away from the predetermined photographing position, the distance between the feature point and the reference point becomes large.

Therefore, by displaying an image of the distance between the feature point and the reference point on the display device by the notifying means, that is, by notifying according to the distance between the feature point and the reference point, the subject to be photographed is notified by the notification. Since it is possible to easily know from the image how far the face is from the predetermined shooting position, it is possible to recognize the shooting position suitable for shooting. As a result, the person to be photographed can easily execute the movement of the face to a predetermined photographing position.

  The method for determining the position of the reference point described above is not particularly limited. For example, a position where a feature point exists in an average face model may be used as a reference point, or a photographed image obtained by photographing a face of a person to be photographed at a predetermined photographing position is acquired in advance, and the photographed image is obtained. The position of the feature point on the face may be used as the reference point.

  Incidentally, as described above, feature points and reference points are set at a plurality of locations. If only one feature point is set, even if the feature point overlaps the reference point, it cannot be determined whether or not the face direction in the captured image is an appropriate direction. It cannot be determined that the captured image was captured when the camera was at the shooting position. On the other hand, if two feature points are set, it is possible to acquire a captured image that matches up to two angles of the face position and the pitch, roll, and yaw related to the face orientation. Further, in order to photograph accurately when the face of the person to be photographed is at a predetermined photographing position, it may be configured as in the photographing auxiliary device according to claim 2.

According to a second aspect of the present invention, in the photographing auxiliary device according to the first aspect, the feature points include three points that are not on the same straight line in the photographed image.
In the photographing auxiliary device configured in this way, at least three points on the face that are not in the same straight line are used as feature points, and the distance between the feature point and the reference point is measured. The distance between the feature point and the reference point can be recognized. Therefore, if photographing is performed in a state where these three points are matched, a photographed image that accurately defines the position and orientation of the face can be acquired.

  According to a third aspect of the present invention, in the photographing auxiliary device according to the first or second aspect, the feature points are three points of the left eye head, the right eye head, and the right and left nostrils on the face of the photographed image. It is characterized by being.

  In the imaging assistance apparatus configured as described above, since the distance from the reference point is measured using at least the three points as the feature points, the distance between the feature points and the reference points at the three points can be recognized by the subject. it can. Further, if photographing is performed in a state where these three points are matched, a photographed image that accurately defines the position and orientation of the face can be acquired. In addition, since the positions of the eyes and nostrils are less affected by changes in facial expressions, the position and orientation of the face can be defined without being affected by changes in facial expressions.

The invention described in claim 4 is the imaging assistance device according to any one of claims 1 to 3, wherein the image information includes a line segment including a feature point and a line segment including a reference point. It is characterized by showing an image further superimposed on a plane where feature points and reference points are superimposed.

  In the imaging assistance device configured as described above, in addition to the above-described feature points and reference points, an image including a line segment including them can be displayed on the display device. Therefore, it is possible to easily determine which feature point is separated from the reference point and how the face is moved to a predetermined photographing position by moving the face.

In the above-described photographing auxiliary device, the notification unit may be configured to output voice information corresponding to the distance measured by the distance measurement unit to a voice output device capable of outputting voice.

  In the photographing auxiliary device configured in this way, the distance between the feature point and the reference point can be output as a sound by the sound output device. Therefore, the person to be photographed can easily recognize by voice how far the face is from the predetermined photographing position, and can easily move to the predetermined photographing position.

Note that the above-described auxiliary device may be configured such that the above-described sound information changes the sound to be output according to the magnitude of the distance measured by the distance measuring unit.

In the photographing assistance apparatus configured as described above, the sound changes depending on the measured distance, and thus the person to be photographed can know the distance between the feature point and the reference point by the change in the sound.
According to a ninth aspect of the present invention, in the photographing auxiliary device according to any one of the first to eighth aspects, when the distance measured by the distance measuring means is equal to or less than a predetermined threshold, the photographed image is photographed. It is characterized by comprising determination means for determining that the subject person's face is a photographed image photographed at the above-mentioned predetermined photographing position.

In the photographing auxiliary device configured as described above, since the photographed image captured when the photographer's face is at a predetermined photographing position can be selected from the photographed images acquired by the image acquisition unit, A photographed image photographed when the subject's face is at the predetermined photographing position described above can be reliably acquired.
The notifying unit is configured to change the display mode of the image information when the determining unit determines that the photographed image is captured when the face of the person to be imaged is at a predetermined shooting position. May be.
In addition, when the determination unit determines that the image of the person to be imaged is a photographed image taken when the face of the person to be photographed is at the predetermined photographing position, the display position of the feature point indicated in the image information May be configured to move slightly.

A twelfth aspect of the invention is a program for causing a computer system to execute a processing procedure for causing each of the means according to any of the first to eleventh aspects to function.
A computer system controlled by such a program can constitute a part of the photographing assistance apparatus according to any one of claims 1 to 11 .

  Note that the above-described program is composed of an ordered sequence of instructions suitable for processing by a computer system, and is provided to a photographing auxiliary device and a user who uses this through various recording media and communication lines. Is.

A thirteenth aspect of the present invention is a photographing auxiliary device according to any one of the ninth to eleventh aspects , a model creating means for creating a face shape model, and an object that indicates the face shape model and the face of the subject. And a detection unit that detects a change in the orientation and expression of the face of the person to be photographed using the person image. The model creating means creates a face shape model using the photographed image determined by the determining means as a photographed image photographed when the face of the subject to be photographed is at a predetermined photographing position. .

In the photographing system configured as described above, a face shape model is created using a photographed image photographed when the face of the subject to be photographed is at a predetermined photographing position. Therefore, it is possible to detect a change in the face direction and facial expression of the subject to be photographed with high accuracy.
The photographing system includes a coincidence determining unit that determines whether or not the target person image and the face shape model are equal to or greater than a predetermined reference. The face shape model may be reconstructed when it is determined that they do not match at the reference level or more.
The model creating means in the photographing system has the smallest distance from a plurality of photographed images determined by the judging means as photographed images taken when the face of the person to be photographed is at the predetermined photographing position. You may comprise so that the said face shape model may be created using.

Side view showing the shooting system Block diagram showing the shooting system Flow chart showing processing procedure of facial expression feature amount detection processing Diagram showing a model of a three-dimensional shape The figure which shows the image which shows the hue of the face shape model Conceptual diagram showing a method of calculating an error value RMS indicating the degree of matching between the input image and the face shape model Conceptual diagram showing a method of calculating an error value RMS when the face shape model is changed The flowchart which shows the process sequence of a front image acquisition process Illustration of feature point extraction method The figure which shows the calculation method of the average value of the distance of the feature point and the reference point The figure which shows the screen which is displayed on the display device

Embodiments of the present invention will be described below with reference to the drawings.
(1) Overall Configuration The photographing system 1 is used by being mounted on a vehicle 2, and includes a camera 10, an information input / output device 20, and a photographing auxiliary device 30, as shown in FIG.

  Among these, the camera 10 continuously captures the face image of the driver 3, and is arranged from the front direction of the driver 3 in the vehicle 2 toward the driver 3 on the dashboard of the vehicle 2. ing.

  As shown in FIG. 2, the information input / output device 20 includes a display device 22 for displaying various image information to the driver 3, a speaker 24 for outputting various guide sounds, and the like, And an operation button 26 for inputting a command by an external operation. The information input / output device 20 displays various images on the display device 22 and outputs sound through the speaker 24 based on a command from the photographing assisting device 30. Further, the content of the external operation performed with the operation button 26 is output to the photographing auxiliary device 30.

  The display device 22 is disposed toward the driver 3 in the vicinity of the camera 10 as shown in FIG. Therefore, the driver 3 can easily confirm the image information displayed on the display device 22 with the face facing the camera 10.

The imaging assistance device 30 includes an image capture board 32, a storage device 34, and a control device 36.
The image capture board 32 temporarily stores photographed image data that is a photographed image data of the face of the driver 3 photographed by the camera 10, and provides it to the control device 36 in response to a command from the control device 36.

The storage device 34 includes a hard disk drive capable of storing various data.
The control device 36 is composed mainly of a well-known microcomputer comprising a CPU, ROM, RAM, I / O and a bus line connecting these components, and an expression to be described later according to a program stored in the ROM. A feature amount detection process and a front image acquisition process are executed.
(2) Processing by Imaging Assistant Device 30 Various processes executed by the control device 36 of the imaging assistant device 30 will be described below.
(2.1) Facial Expression Feature Quantity Detection Processing First, the processing procedure of facial expression feature quantity detection processing will be described with reference to FIG. This process is continuously executed while the photographing system 1 is activated.

In this process, first, captured image data of the face of the driver 3 captured by the camera 10 and stored in the image capture board 32 is acquired (S1).
Next, using the captured image data acquired in S1, fitting with the face shape model stored in the storage device 34 is performed, and the change in facial orientation and facial expression of the driver 3 (hereinafter simply referred to as facial expression feature amount). Is also detected (S2). Details will be described below.

The face shape model is obtained by superimposing an image showing the color of a human face on a three-dimensional model representing the shape of the human face.
First, a three-dimensional shape model that represents the shape of a human face will be described. As shown in FIG. 4, the three-dimensional shape model includes a basic shape (base shape) s0 (see FIG. 4 (a)) in which the front face is represented by a plurality of triangular meshes, and a basic shape s0. N (n is a natural number) shape vector (i = 1, 2, 3,..., N) indicating face position / face orientation / expression change (FIGS. 4 (b), (c), (See (d)) is set in advance, and the shape of the human face is expressed by calculating the basic shape s0 and n shape vectors si using the equation (1) (the mesh in FIG. 6). s).

  In the above formula (1), pi (i = 1, 2, 3...) Is a superposition coefficient. Further, the shape vector si (i = 1, 2, 3...) Is a vector starting from the vertexes of the mesh constituting the basic shape s0 (hereinafter also referred to as mesh vertices), and all the vertices of the basic shape s0. A shape vector si is set for (in FIGS. 4B, 4C, and 4D, shape vectors s1, s2, and s3 are displayed only for some mesh vertices).

  Next, an image showing the color of a human face will be described. As shown in FIG. 5, the face shape model includes an image (base appearance image) A0 (x) (hereinafter referred to as a basic hue image A0 (x)) indicating a basic hue of the front face, and a fitting of the hue. In this case, m (m is a natural number) appearance images Ai (x) (i = 1, 2, 3,..., M) to be superimposed on the basic hue image A0 (x) are set in advance. The hue of a human face is expressed by calculating the basic hue image A0 (x) and m number of hue images Ai (x) using the equation (2) (A (x) in FIG. 6). ).

Here, x indicates a two-dimensional coordinate in the basic shape s0. In addition, λi (i = 1, 2, 3...) In the equation (2) is an overlay coefficient.
Then, as shown in FIG. 6, the three-dimensional shape and hue of the face shape model are changed, and the changed face shape model M (W (x; p)) and the facial expression feature quantity detection target person are photographed. Fitting with the input image I which is the obtained image is performed, and the degree of coincidence is determined.

  The following formula (3) is a formula for calculating the error value RMS indicating the degree of matching between the input image and the face shape model. M (W (x; p)) is an image when the hue image in the mesh s determined by the equation (1) is A (x), and I (W (x; p)) is the mesh s in the input image I. It is a shade image inside. The mesh s can be moved on the input image I while its shape is deformed by the superposition coefficient pi. Note that the shape vector si includes vectors such as vertical movement, horizontal movement, enlargement / reduction, and rotation in addition to vectors that change the facial expression and face orientation.

  In order to detect the facial expression feature quantity from the input image I, the superposition coefficients pi and λi that minimize the RMS in the equation (3) are calculated. That is, while changing the superposition coefficients pi and λi to create images M (W (x; p)) of various shapes and shades, the image I ( W (x; p)) is searched on the input image I.

  Then, the overlay coefficients pi and λi that minimize the RMS are calculated. Since the value of pi and λi indicates how the face shape model has changed, it is possible to detect the facial expression feature amount of the detection target person.

  Next, it is determined whether or not the face shape model needs to be updated (S3). Here, it is determined whether or not the minimum value of RMS calculated by the above equation (3) is below a predetermined threshold. That the minimum value of RMS is large means that the captured image (input image I) based on the captured image data acquired in S1 does not match the face shape model with high accuracy. If the expression is different from the facial expression, the RMS may be minimized, and there is a possibility that the detection accuracy of the facial expression feature amount is lowered. Therefore, when the minimum value of RMS exceeds a predetermined threshold value, it is necessary to change the face shape model and increase detection accuracy.

  If it is not determined in S3 that the face shape model needs to be changed (S3: NO), the facial expression feature quantity can be detected accurately even if the face shape model is used without being changed, so that the face shape model is not changed. The process returns to S1. On the other hand, if it is determined in S3 that the face shape model needs to be changed (S3: YES), the front image acquisition process is performed to acquire front-captured image data obtained by accurately capturing the driver 3 from the front. (S4) Based on the acquired front image data, the face shape model is reconstructed (S5). Details will be described below.

  When facial expression features of a plurality of detection subjects are acquired using a single facial shape model, it is common to create a facial shape model using an average of multiple facial expressions as A0 (x). (See A0 (x) in FIG. 6). However, in this case, since the input image I and the face shape model M are not images of the same person, the RMS becomes large and an accurate facial expression feature amount may not be captured.

  Therefore, in order to solve this problem, s0 and A0 are replaced with information on the subject of the input image I. As shown in FIG. 7, s′0 and A0 ′ are created from the front-captured image data acquired in S4, and a face shape model M ′ (W (x; p)) with the subject's characteristics added is constructed. Thus, RMS can be reduced and an accurate facial expression feature amount can be captured.

The face shape model created in this way is used when fitting is performed in the subsequent S2. After S5, the process returns to S1.
(2.2) Front Image Shooting Process Next, the processing procedure of the front image shooting process will be described with reference to FIG. This process is executed in S4 of the facial expression detection process.

In this process, first, captured image data of the face of the driver 3 captured by the camera 10 and stored in the image capture board 32 is acquired (S11).
Next, feature points on the face in the photographed image based on the photographed image data acquired in S11 are extracted (S12). Here, as shown in FIG. 9A, the positions on the image plane of the left and right eyes and the midpoint of the left and right nostrils when the captured image is represented on the xy plane are extracted.

  Specifically, first, binarization processing is performed to divide the photographed image data into a black portion and a white portion (for example, the technique described in Japanese Patent Application No. 03-220633 may be used), and the image data The positions of the left eye, right eye, left nostril, and right nostril on the xy plane in the image based on the image are detected (for example, the technique described in Japanese Patent Application No. 08-101904 may be applied). Then, the position of the midpoint of the left and right nostrils is calculated from the positions of the left and right nostrils. Note that the feature point extraction method is not limited to the above-described method, and any method may be used for extraction.

  Next, using the feature points on the face extracted in S12 and the reference points of the middle points of the left and right eyes and the left and right nostrils stored in advance in the storage device 34, the average of the distances between the corresponding points is calculated. A value (hereinafter simply referred to as an error) is calculated (S13). As shown in FIG. 10, the average distance here is a distance L1 between the feature point related to the left eye and the reference point, a distance L2 between the feature point related to the right eye and the reference point, and left and right This is an arithmetic mean value of the distance L3 between the feature point and the reference point regarding the midpoint of the nostril.

  The reference point described above is set at a position where the feature point exists in the average face model, but the face of the driver 3 is photographed in advance with high accuracy and extracted from the photographed image. A feature point may be used as a reference point.

  Next, image information of an image obtained by superimposing a current position display line α (hereinafter simply referred to as line α) and a reference position display line β (hereinafter simply referred to as line β) on a plane is output to the display device 22. (S14).

  As shown in FIG. 9B, the line α includes a line segment connecting the left and right eyes, the midpoint of the left and right eyes, and the midpoint of the left and right nostrils with respect to the captured image data acquired in S11. It is a line segment consisting of connected line segments. In addition, the line β, using the reference point described above, similarly to the line α, a line segment connecting the left and right eyes, a line segment connecting the midpoint of the left and right eyes and the right and left nostrils, It is a line segment consisting of

  In response to this processing of S14, the display device 22 displays the display screen shown in FIGS. 11 (a) and 11 (b). On the display screen, an instruction to overlap the lines α and β by moving the face is also displayed.

  The line α and the line β can be identified on the display screen of the display device 22 by changing the color, the shape of the line, the presence or absence of blinking, and the like. In this embodiment, the line α is displayed as a green broken line, and the line β is displayed as a blue solid line.

  The display mode of the line α changes depending on whether or not the error calculated in S13 is equal to or less than a predetermined threshold value. The threshold here is a value that serves as a reference for determining that the face of the driver 3 has moved to a predetermined photographing position when the error is equal to or smaller than the threshold.

  However, when the reference point is determined using an average face model, the position of the facial feature point of the driver 3 differs for each driver 3, so the minimum error value for each driver 3 is Change. For this reason, if the threshold value is set as a constant value, the error can be reduced to the threshold value or less relatively easily in the case of the driver 3 having a feature point close to the average face model. If the feature point is located at a position that is significantly different from the feature point of a simple face model, it may be difficult to make it less than the threshold value, or it may be impossible to make it less than the threshold value.

Therefore, this threshold value is appropriately changed according to the average value of errors calculated by executing the process of S13 a predetermined number of times.
Returning to the description of the display screen displayed on the display device 22. In this embodiment, when the error exceeds a predetermined threshold value, the display of the line α does not change. However, when the error is equal to or lower than the threshold value, the color of the line α is changed to red so that the face of the driver 3 becomes a predetermined value. The driver 3 is notified that it has been determined that the camera has moved to the shooting position (see FIG. 11B).

  Next, audio information is output to the speaker 24 (S15). As a result, the speaker 24 outputs a guide voice. Here, an intermittent sound having a constant volume, height, and tone color is output, but the sound is output so that the interval between the sounds becomes shorter as the error calculated in S13 is smaller.

  Next, it is checked whether or not the error calculated in S13 is equal to or smaller than the threshold value described above. If the error is equal to or smaller than the threshold value, the captured image data acquired in S11 is stored in the storage device 34 (S16). At that time, together with the captured image data, the error value calculated in S13 is stored in association with the captured image data.

  Next, it is determined whether or not the number of captured image data stored in the storage device 34 in S16 exceeds a predetermined number (S17). If the predetermined number has not been reached (S17: NO), the process returns to S11, and the captured image data is acquired again. On the other hand, if the predetermined number has been reached (S17: YES), among the photographed image data stored in the storage device 34, the photographed image data with the smallest error is used for the front photographed image used for the reconstruction of the face shape model. Select as data (S18). After S18, the process returns to the facial expression feature amount detection process.

Since the loop of S11 to S17 described above is performed at a short time interval (for example, 0.1 s), the position of the line α moves in real time as the driver 3 moves the face with respect to the camera 10. To do.
(3) Effects of the Invention In the imaging system 1 configured as described above, when the accuracy of the fitting between the face shape model and the captured image is poor, a front captured image obtained by accurately capturing the face of the driver 3 from the front is obtained. Since the facial shape model can be reconstructed by using it, the facial expression feature quantity can be captured with higher accuracy.

  In acquiring the above-described front shot image, the display device 22 displays the feature points and predetermined reference points extracted from the shot image, and the lines α and β based on them. The driver 3 can confirm from the image how far the face photographed by the camera 10 is from the position of the face suitable for photographing and the direction of the face (predetermined photographing position). A suitable face position and face orientation can be easily recognized.

  Further, since the above-mentioned image is displayed on the display device 22 and at the same time, a sound corresponding to the magnitude of the error is output from the speaker 24, the driver 3 can also capture the face photographed by the camera 10 by sound. It is possible to know how far away from a predetermined shooting position.

  As described above, the driver 3 can easily recognize the face position and the face direction suitable for photographing, so that the driver 3 moves the face to the recognized face position and matches the face direction. It is possible to take a highly accurate front shot image.

  In the photographing system 1, the left and right eyes and the midpoints of the left and right nostrils are defined as feature points and reference points, and the position and orientation of the face can be defined based on these points. Note that, by defining the three points that are not in the same straight line in this way, it is possible to acquire captured image data that accurately defines the position and orientation of the face. In addition, since the eyes and nostrils are places where the position change due to the expression change is small, the position and orientation of the face can be defined without being affected by the expression change.

Further, in the photographing system 1, when the error becomes a predetermined threshold value or less, the photographed image data can be selected as the data of the front photographed image, so that highly accurate front photographed image data is reliably acquired. can do.
(4) Modifications While the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and can take various forms as long as they belong to the technical scope of the present invention. Needless to say.

  For example, when the error between the feature point and the reference point is equal to or less than a predetermined threshold, the display position of the line α on the display device 22 is slightly moved regardless of the face position and face direction of the driver 3. You may comprise as follows. Specifically, when the error calculated in S13 of the front image acquisition process is equal to or smaller than the threshold value, the position of the line α in the image information output in S14 is slightly moved.

  In the imaging system 1 configured as described above, when the driver 3 moves his face and the error becomes equal to or less than the threshold value, the line α slightly moves, so the driver 3 again matches the line α and the line β. Therefore, the face is moved slightly. As a result, it is possible to acquire many patterns of captured image data in the vicinity of a position that is equal to or lower than the threshold value, so that it is possible to increase the probability of capturing a highly accurate front-captured image.

  Further, in the above-described embodiment, a configuration in which a predetermined number of photographed image data whose error is equal to or smaller than a predetermined threshold is stored in S16 of the front image acquisition process, and appropriate photographed image data is selected from among them is selected in S18.

  However, in the case where captured image data having an error equal to or less than a predetermined threshold value is acquired without storing a predetermined number of captured image data in S16, the captured image data may be used as front captured image data. With the imaging system 1 configured as described above, it is only necessary to acquire one captured image data with an error equal to or less than the threshold value, and thus the front image acquisition process can be quickly completed.

  Further, in the above embodiment, in S13 of the front image acquisition process, an arithmetic average of the distances between the feature points and the reference points is obtained, and the face of the driver 3 is determined to be a predetermined shooting position based on the value. Although the configuration for determining whether or not the above has been exemplified, the above determination may be performed using other values. For example, the structure which performs the said determination based on the geometric mean of the distance between each feature point and a reference point can be considered.

  Moreover, in the said Example, although the structure which displays on the display apparatus 22 the image which superimposed the line (alpha) and the line (beta) on the plane in S14 of front image acquisition process was illustrated, the imaging | photography acquired at S11 immediately before It may be configured to further superimpose and display an image.

With the imaging system 1 configured in this way, it is possible to recognize to which part of the face the line α corresponds, so that it becomes easy to understand image information sensuously.
Moreover, in the said Example, although the display apparatus 22 and the speaker 24 illustrated the structure which alert | reports so that the driver | operator 3 can recognize the difference | error calculated in S13, driving | operation using the display apparatus 22 and the speaker 24 was illustrated. You may comprise so that the alerting | reporting which guides the direction which turns a face to the person 3 may be performed. For example, a configuration in which the direction in which the face is directed is displayed with an arrow on the display screen is conceivable.

  In the above-described embodiment, the line α displayed on the display screen is exemplified by a configuration in which the color changes from green to red when the error is equal to or smaller than the threshold value. However, the display mode of the line α is changed in other configurations. There may be. For example, a configuration in which the line α changes from green to transparent when the error is equal to or smaller than a threshold value, and a configuration in which the shape of the line α (bold line and thin line, or a solid line and a dotted line) changes when the error is equal to or smaller than the threshold value. Further, the display mode of the line β may be changed when the error is equal to or less than the threshold value.

  In the above embodiment, the lines α and β are composed of a line segment connecting the left and right eyes on the face, and a line segment connecting the left and right eyes midpoint and the left and right nostrils midpoint. Although illustrated, it may be composed of other line segments. For example, it is conceivable that a triangle connecting three points including the left and right eyes and the right and left nostril midpoints is defined as a line α and a line β.

  In addition, the operation button 26 may be used to allow the driver 3 to select the notification method (display mode and display content on the display device 22, content of audio output by the speaker 24, etc.).

  In the above-described embodiment, the configuration in which the reconstruction of the face shape model is started when the minimum value of the RMS value falls below a predetermined threshold in S3 of the facial expression feature amount detection process is illustrated. It may be configured to start reconstruction of the face shape model in accordance with the operation by.

  Moreover, in the said Example, although the point used as a feature point and a reference point illustrated the structure which is a right-and-left both eyes head and a right-and-left nostril midpoint, other points may be sufficient. In that case, the point used as the feature point is preferably a point with little movement of the position due to the change of facial expression, such as the eyes and nostrils.

  Moreover, in the said Example, although the structure which has the display apparatus 22 arrange | positioned facing the driver | operator 3's face direction was illustrated, it replaced with the display apparatus 22 and provided with the virtual image display apparatus which displays a virtual image on a windshield. The image display in S14 of the front image acquisition process may be a virtual image display.

  In the above-described embodiment, the feature point and the reference point are determined on the captured image plane and the error is measured. However, the direction in which the driver 3 faces is the three axes of roll, pitch, and yaw. You may comprise so that it may measure as an angle parameter and may measure an error with a predetermined reference angle.

  In that case, the intermittent sound output from the speaker 24 is set to a different frequency for each axis of roll, pitch, and yaw, and an intermittent sound with an interval corresponding to the angle error is output for each axis. The angular error for each axis can be recognized by voice. Further, by changing the display color of the display object (the entire display screen, a partial area on the display screen, or the above-described line α) displayed by the display device 22, each axis can be changed. You may comprise so that an angle error may be recognized. For example, a hue corresponding to each axis of roll, pitch, and yaw is determined (for example, roll is red, pitch is green, and yaw is blue), and each color is set at a density corresponding to an angle error of the corresponding axis. It is conceivable to synthesize and output the synthesized color as the display color of the display object.

  Specifically, when the error between the roll angle and the pitch angle is large and the error in the yaw angle is small, a display object with a color that combines dark red and green with light blue is displayed, and only the yaw angle has an error. In some cases, the display object is displayed in blue, and when the face is located at a predetermined shooting position with no error on all axes, the display object is displayed in white. Note that the relationship between the angle error and the shade and the hue corresponding to each axis are not limited to those described above.

  Moreover, in the said Example, although the structure which alert | reports so that the driver | operator 3 can recognize the difference | error calculated by S13 by the display apparatus 22 and the speaker 24 was illustrated, only one of the display apparatus 22 and the speaker 24 is shown. May be configured to perform notification.

DESCRIPTION OF SYMBOLS 1 ... Shooting system, 2 ... Vehicle, 3 ... Driver, 10 ... Camera, 20 ... Information input / output device, 22 ... Display device, 24 ... Speaker, 26 ... Operation button, 30 ... Shooting auxiliary device, 32 ... Image capture board 34 ... Storage device, 36 ... Control device

Claims (15)

Image acquisition means for acquiring a captured image showing the face of the person to be imaged;
Extraction means for extracting a plurality of feature points on the face in the captured image acquired by the image acquisition means;
The shooting is performed so that the distance between the feature point extracted by the extraction unit and the feature point exists corresponding to each of the feature points, and the distance from the feature point is minimized when the face of the person to be shot is at a predetermined shooting position. A distance measuring means for measuring a distance between a predetermined reference point on the image, and
An imaging assistance device comprising: a notification unit that outputs image information indicating an image obtained by superimposing the feature point and the reference point on a plane with respect to a display device capable of displaying an image . The imaging assistance device according to claim 1, wherein the feature points include three points that are not on the same straight line in the captured image. The imaging assisting device according to claim 1, wherein the feature points are three points of a left eye head, a right eye head, and a midpoint of left and right nostrils on the face of the photographed image. The image information indicates an image obtained by further superimposing a line segment including the feature point and a line segment including the reference point on a plane on which the feature point and the reference point are superimposed. The photographing auxiliary device according to any one of claims 1 to 3 . The notification means is further for outputting capable sound output device sound, one of claims 1, characterized in that outputs audio information corresponding to the distance measured by the distance measuring means according to claim 4 The photographing auxiliary device according to the above. 6. The photographing assistance apparatus according to claim 5 , wherein the sound information changes sound to be output according to the magnitude of the distance measured by the distance measuring unit.   The reference point is a position where a feature point exists in an average face model
  The photographing assisting device according to any one of claims 1 to 6, wherein   The reference point is a position of a feature point on the face in a photographed image obtained by photographing the face of the subject to be photographed in advance at the predetermined photographing position.
  The photographing assisting device according to any one of claims 1 to 6, wherein Determination that the captured image is determined to be a captured image taken when the face of the person to be photographed is at the predetermined photographing position when the distance measured by the distance measuring unit is equal to or smaller than a predetermined threshold. The photographing auxiliary apparatus according to claim 1, further comprising: means.   The notification means changes the display mode of the image information when the determination means determines that the photographed image is taken when the face of the person to be photographed is at the predetermined photographing position.
  The photographing auxiliary apparatus according to claim 9.   The notification unit displays the feature point indicated by the image information when the determination unit determines that the image of the person to be imaged is a photographed image taken when the face is the predetermined photographing position. Move
  The photographing assisting apparatus according to claim 9 or 10, wherein the photographing assisting apparatus is characterized. The program for making a computer system perform the process sequence for functioning as each means in any one of Claims 1-11 . An imaging assistance device according to any one of claims 9 to 11 ,
Model determination means for generating the face shape model using the captured image determined by the determination means as a captured image captured when the face of the person to be imaged is at the predetermined imaging position;
Detecting means for detecting a change in facial orientation and facial expression of the subject to be photographed using the face shape model created by the model creating means and a subject image showing the face of the subject to be photographed; An imaging system characterized by comprising:   A match determining means for determining whether or not the target person image and the face shape model match at or above a predetermined reference;
  The model creation unit reconstructs the face shape model when the match determination unit determines that the match does not exceed the predetermined reference.
  The imaging system according to claim 13.   The model creating means uses the one having the smallest distance from a plurality of the photographed images determined by the judging means to be photographed when the face of the person to be photographed is at the predetermined photographing position. To create the face shape model
  The imaging system according to claim 13 or claim 14, characterized in that: