JP7318352B2 - Image processing device, monitoring device, control system, image processing method, and program - Google Patents

Description

The present invention relates to an image processing device, a monitoring device, a control system, an image processing method, and a program.

Patent Literature 1 below discloses a robot device used as a service providing device capable of switching to an appropriate service according to the situation of a target (person) to whom the service is provided.

The robot device is equipped with a first camera, a second camera, and an information processing device including a CPU. The CPU includes a face detection unit, an attribute determination unit, a person detection unit, a person position calculation unit, and It is equipped with a movement vector detection unit.

According to the above-described robot device, when the target of service provision is a group of persons who have established relationships such as mutual communication, the first service of providing information based on close exchanges is provided. to decide. On the other hand, if the target of service provision is a group of people whose relationship such as mutual communication is unknown, the second service provides information unilaterally without any interaction. decide to do As a result, appropriate services can be provided according to the situation of the service provider.

[Problems to be solved by the invention]
In the robot device, the face detection unit is configured to detect a person's face using the first camera, and known technology can be used for the face detection.
However, in the conventional face detection technology, when a part of the facial organs such as eyes, nose, mouth, etc. is missing or greatly deformed due to injury, etc., there is a large mole, wart, or body decoration such as a tattoo on the face. Such a specific individual (in other words, age difference, gender, and person There is a problem that the accuracy of face detection for a specific individual having features different from the facial features of general people common regardless of species etc. is lowered.

JP 2014-14899 A

Means to solve the problem and its effect

The present invention has been made in view of the above problems, and provides an image processing device, a monitoring device, a control system, an image processing method, and a program that can improve the accuracy of face sensing for a specific individual as described above. It is intended to

To achieve the above object, an image processing device (1) according to the present disclosure is an image processing device that processes an image input from an imaging unit,
a facial feature amount storage unit that stores a specific individual's facial feature amount and a normal facial feature amount as learned facial feature amounts that have been trained to detect a face from the image;
a face detection unit that detects a face area while extracting a feature amount for detecting a face from the image;
a specific individual determination unit that determines whether or not the face in the face region is the face of the specific individual using the detected feature quantity of the face region and the face feature quantity of the specific individual;
a first face image processing unit that performs face image processing for a specific individual when the specific individual determining unit determines that the face is the face of the specific individual;
and a second face image processing unit that performs normal face image processing when the specific individual determining unit determines that the face is not that of the specific individual.

According to the image processing apparatus (1), the facial feature amount of the specific individual and the normal facial feature amount (in other words, the specific individual and the specific individual determination unit stores the feature amount of the face region detected by the face detection unit and the facial feature amount of the specific individual. is used to determine whether or not the face in the face area is the face of the specific individual. By using the face feature amount of the specific individual, it is possible to accurately determine whether or not it is the face of the specific individual.
Further, when it is determined to be the face of the specific individual, the face image processing of the specific individual can be accurately performed by the first face image processing section. On the other hand, when it is determined that the face is not the face of the specific individual, in other words, the face is a normal face (the face of a person other than the specific individual), the normal face image processing is performed by the second face image processing unit. It can be executed with high accuracy. Therefore, whether it is the specific individual or a normal person other than the specific individual, the sensing of each face can be performed with high accuracy.

Further, in the image processing device (2) according to the present disclosure, in the image processing device (1), the specific individual determining unit determines a correlation between the feature quantity extracted from the face region and the face feature quantity of the specific individual. is calculated, and whether or not the face in the face area is the face of the specific individual is determined based on the calculated index.

According to the image processing device (2), an index indicating a correlation between the feature quantity extracted from the face region and the face feature quantity of the specific individual is calculated, and the face region is calculated based on the calculated index. is the face of the specific individual. Thereby, it is possible to efficiently determine whether or not the face in the face area is the face of the specific individual based on the index. The index may be an index value indicating that the higher the value, the higher the relationship, such as a correlation coefficient, the reciprocal of a squared error, or an index extracted from the face region. It may be an index value or the like indicating the degree of similarity of the relationship between the feature amount obtained and the facial feature amount of the specific individual.

Further, in the image processing device (3) according to the present disclosure, in the image processing device (2), when the index is larger than a predetermined threshold, the face in the face region is the face of the specific individual. and if the index is equal to or less than the predetermined threshold value, it is determined that the face in the face area is not the face of the specific individual.

According to the image processing device (3), if the index is greater than a predetermined threshold, the face in the face region is determined to be the face of the specific individual, and if the index is equal to or less than the predetermined threshold, the It is determined that the face in the face area is not the face of the specific individual. The processing efficiency of the determination can be further improved by the processing of comparing the index and the predetermined threshold value.

Further, the image processing device (4) according to the present disclosure is any one of the above image processing devices (1) to (3), wherein the specific individual determination unit determines, based on the determination result for one frame of the image, the It is characterized by determining whether or not the face in the face area is the face of the specific individual.

According to the image processing device (4), it is determined whether or not the face in the face area is the face of the specific individual based on the result of determination for one frame of the image. can be improved.

Further, the image processing device (5) according to the present disclosure is any one of the image processing devices (1) to (3), wherein the specific individual determination unit determines the It is characterized by determining whether or not the face in the face area is the face of the specific individual.

According to the image processing device (5), it is determined whether or not the face in the face region is the face of the specific individual based on the results of determination for a plurality of frames of the image. can increase

Further, the image processing device (6) according to the present disclosure, in any one of the image processing devices (1) to (5), includes face detection processing, face direction estimation processing, gaze direction estimation processing, and eye open/close detection processing.

According to the image processing device (6), the face image processing includes at least one of face detection processing, face direction estimation processing, line-of-sight direction estimation processing, and eye open/close detection processing. It is possible to accurately perform processing for estimating and detecting various facial behaviors of the specific individual or a (normal) person other than the specific individual.

Further, the monitoring device (1) according to the present disclosure includes any of the image processing devices (1) to (6), an imaging unit that captures an image to be input to the image processing device, and image processing by the image processing device. and an output unit for outputting information based on the information.

According to the monitoring device (1), not only the face of the ordinary person but also the face of the specific individual can be monitored with high accuracy, and information based on the image processing can be output from the output unit. Therefore, it is possible to easily construct a monitoring system using the information.

Further, a control system (1) according to the present disclosure is communicably connected to the monitoring device (1) and the monitoring device, and executes a predetermined process based on the information output from the monitoring device. It is characterized by comprising the above control device.

According to the above control system (1), it is possible to cause one or more of the control devices to execute a predetermined process based on the information output from the monitoring device. Therefore, it is possible to construct a system that can use not only the monitoring results of the ordinary person but also the monitoring results of the specific individual.

Further, the control system (2) according to the present disclosure is the above control system (1),
the monitoring device is a device for monitoring a driver of a vehicle;
The control device is characterized by including an electronic control unit mounted on the vehicle.

According to the control system (2), even if the driver of the vehicle is the specific individual, the face of the specific individual can be monitored with high accuracy, and based on the monitoring results, the electronic It is possible to cause the control unit to appropriately execute predetermined control. As a result, it is possible to construct a highly safe vehicle-mounted system that allows even the specific individual to drive with peace of mind.

Further, an image processing method according to the present disclosure is an image processing method for processing an image input from an imaging unit,
a face detection step of detecting a face region while extracting face feature amounts from the image;
using the feature quantity of the face region detected by the face detection step and the learned face feature quantity of the specific individual that has undergone learning to detect the face of the specific individual; a specific individual determination step of determining whether or not is the face of the specific individual;
a first face image processing step of performing face image processing for a specific individual when the face is determined to be that of the specific individual by the specific individual determination step;
and a second face image processing step of performing normal face image processing when it is determined by the specific individual determination step that the face is not that of the specific individual.

According to the above image processing method, in the specific individual determination step, the face in the face region is determined using the feature quantity of the face region detected in the face detection step and the face feature quantity of the specific individual. It is determined whether or not it is the face of the specific individual. By using the face feature amount of the specific individual, it is possible to accurately determine whether or not it is the face of the specific individual.
Further, when it is determined that the face is the face of the specific individual, the face image processing of the specific individual can be performed with high accuracy by the first face image processing step. On the other hand, when it is determined that the face is not the face of the specific individual, in other words, the face is a normal face that is not the specific individual, the normal face image processing is performed with high accuracy by the second face image processing step. can be done. Therefore, whether it is the specific individual or a normal person other than the specific individual, the sensing of each face can be performed with high accuracy.

A program according to the present disclosure is a program for causing at least one or more computers to process an image input from an imaging unit,
to said at least one computer;
a face detection step of detecting a face region while extracting face feature amounts from the image;
using the feature quantity of the face region detected by the face detection step and the learned face feature quantity of the specific individual that has undergone learning to detect the face of the specific individual; a specific individual determination step of determining whether or not is the face of the specific individual;
a first face image processing step of performing face image processing for a specific individual when the face is determined to be that of the specific individual by the specific individual determination step;
and a second face image processing step of performing normal face image processing when it is determined by the specific individual determining step that the face is not that of the specific individual.

According to the above program, the at least one computer uses the feature quantity of the face region and the face feature quantity of the specific individual to determine whether the face in the face region is the face of the specific individual. Therefore, it is possible to accurately determine whether or not the face is the face of the specific individual.
Further, when it is determined that the face is the face of the specific individual, the face image processing of the specific individual can be performed with high accuracy. On the other hand, when it is determined that the face is not the face of the specific individual, in other words, it is determined to be a normal face that is not the specific individual, the normal face image processing can be performed with high accuracy. Therefore, it is possible to construct a device or system capable of accurately sensing the face of each of the specific individual and a normal person other than the specific individual. The program may be a program stored in a storage medium, a program that can be transferred via a communication network, or a program that is executed via a communication network. .

1 is a schematic diagram showing an example of an in-vehicle system including a driver monitoring device according to an embodiment of the present invention; FIG. 1 is a block diagram showing an example hardware configuration of an in-vehicle system including a driver monitoring device according to an embodiment; FIG. 2 is a block diagram showing an example functional configuration of an image processing unit of the driver monitoring device according to the embodiment; FIG. 4 is a flowchart showing an example of processing operations performed by an image processing unit of the driver monitoring device according to the embodiment; 7 is a flowchart showing an example of a specific individual determination processing operation performed by an image processing unit of the driver monitoring device according to the embodiment; 9 is a flowchart showing another example of the specific individual determination processing operation performed by the image processing unit of the driver monitoring device according to the embodiment;

Embodiments of an image processing device, a monitoring device, a control system, an image processing method, and a program according to the present invention will be described below with reference to the drawings.
INDUSTRIAL APPLICABILITY The image processing apparatus according to the present invention can be widely applied to, for example, apparatuses and systems for monitoring objects such as people using cameras. The image processing device according to the present invention is, for example, a device or system for monitoring a driver (operator) of various moving bodies such as a vehicle, or operates or monitors various facilities such as machines and devices in a factory. It can also be applied to a device or system for monitoring a person who performs a predetermined work or the like.

[Application example]
FIG. 1 is a schematic diagram showing an example of an in-vehicle system including a driver monitoring device according to an embodiment. In this application example, an example in which an image processing device according to the present invention is applied to a driver monitoring device 10 will be described.

The in-vehicle system 1 includes a driver monitoring device 10 that monitors the state of the driver 3 of the vehicle 2 (for example, behavior of the face), and one or more ECUs (Electronic Control Units) that control driving, steering, or braking of the vehicle 2. ) 40 and one or more sensors 41 for detecting the state of each part of the vehicle or the state of the surroundings of the vehicle, which are connected via a communication bus 43 . The in-vehicle system 1 is configured as an in-vehicle network system that communicates according to CAN (Controller Area Network) protocol, for example. In addition, other communication standards than CAN may be adopted as the communication standard of the in-vehicle system 1 . The driver monitoring device 10 is an example of the "monitoring device" of the present invention, and the in-vehicle system 1 is an example of the "control system" of the present invention.

The driver monitoring device 10 includes a camera 11 for capturing an image of the face of the driver 3, an image processing unit 12 for processing an image input from the camera 11, and information based on image processing by the image processing unit 12 via a communication bus 43. and a communication unit 16 that performs processing for outputting to a predetermined ECU 40 via the communication unit 16 . The image processing unit 12 is an example of the "image processing device" of the present invention. The camera 11 is an example of the "imaging section" of the present invention.

The driver monitoring device 10 detects the face of the driver 3 from the image captured by the camera 11, and detects the behavior of the detected face of the driver 3, such as the direction of the face, the direction of the line of sight, or the state of opening and closing the eyes. The driver monitoring device 10 may determine the state of the driver 3, such as looking ahead, looking aside, dozing off, looking backwards, lying down, etc., based on these facial behavior detection results. In addition, the driver monitoring device 10 outputs a signal based on the state determination of the driver 3 to the ECU 40, and the ECU 40 performs attention or warning processing to the driver 3 or operation control (for example, deceleration control) of the vehicle 2 based on the signal. control, or guidance control to the road shoulder, etc.).

One of the purposes of the driver monitoring device 10 is to improve the accuracy of face sensing for a specific individual.

In the conventional driver monitoring device, the driver 3 of the vehicle 2 has, for example, a part of facial organs such as eyes, nose, mouth, etc. missing or greatly deformed due to injury, or a large mole or wart on the face. Accuracy of detecting a face from an image captured by a camera when the arrangement of the facial organs deviates from the average position due to body decorations such as tattoos or diseases such as genetic diseases. There was a problem that the

In addition, when the face detection accuracy decreases, the processing after the face detection such as the face direction estimation processing is not performed properly. Moreover, there is a problem that various controls that should be executed by the ECU 40 based on the state determination may not be performed appropriately.

In order to solve such a problem, the driver monitoring device 10 according to the embodiment can detect a specific individual, in other words, a common general person regardless of differences in age, gender, race, etc. (individual differences). In order to improve the accuracy of face detection for a specific individual who has features different from those of a (ordinary person), the following configuration was adopted.

The image processing unit 12 stores a specific individual's facial feature amount and a normal facial feature amount (in other words, a person other than a specific individual) as learned facial feature amounts that have been trained to detect a face from an image. ) are stored.

The image processing unit 12 performs face detection processing for detecting a face area while extracting a feature amount for detecting a face from an input image of the camera 11 . Then, the image processing unit 12 uses the detected feature quantity of the face region and the face feature quantity of the specific individual to determine whether or not the face in the face region is the face of the specific individual. Specific individual determination processing is performed.

In the specific individual determination process, a correlation coefficient is calculated as an index indicating the relationship between the feature amount extracted from the facial region and the facial feature amount of the specific individual, for example, as an index indicating correlation, and the calculated correlation coefficient is calculated. Whether or not the face in the face area is the face of the specific individual may be determined based on the relational coefficient.
For example, when the correlation coefficient is greater than a predetermined threshold, the face in the face region is determined to be the face of the specific individual, and when the correlation coefficient is less than or equal to the predetermined threshold, the face in the face region is determined to be the face of the specific individual. It may be determined that the face is not the face of the specific individual. In addition, in the specific individual determination process, an index other than the correlation coefficient may be used.

Further, in the specific individual determination process, it may be determined whether or not the face in the face region is the face of the specific individual based on the determination result for one frame of the input image from the camera 11, Whether or not the face in the face area is the face of the specific individual may be determined based on the results of determination for a plurality of frames of the input image from the camera 11 .

In this way, in the driver monitoring device 10, the learned face feature amount of the specific individual is stored in advance in the image processing unit 12, and by using the face feature amount of the specific individual, the face of the specific individual is obtained. It is possible to accurately determine whether or not.

Further, when it is determined by the specific individual determination process that the face is that of the specific individual, the image processing unit 12 executes face image processing for the specific individual, so that the specific individual's face image processing is accurately performed. It becomes possible to
On the other hand, if it is determined that the face is not the face of the specific individual, in other words, the face is a normal face (in other words, a face other than the specific individual), the image processing unit 12 performs normal face image processing. Therefore, the normal face image processing can be performed with high accuracy. Therefore, regardless of whether the driver 3 is a specific individual or a normal person other than the specific individual, it is possible to accurately sense each face.

[Hardware configuration example]
FIG. 2 is a block diagram showing an example of the hardware configuration of the in-vehicle system 1 including the driver monitoring device 10 according to the embodiment.

The in-vehicle system 1 includes a driver monitoring device 10 that monitors the state of the driver 3 of the vehicle 2 , one or more ECUs 40 , and one or more sensors 41 , which are connected via a communication bus 43 . One or more actuators 42 are connected to the ECU 40 .

The driver monitoring device 10 includes a camera 11, an image processing unit 12 that processes an image input from the camera 11, and a communication unit 16 that exchanges data and signals with an external ECU 40 or the like. there is

The camera 11 is a device that captures an image including the face of the driver 3 sitting in the driver's seat, and includes, for example, a lens unit, an imaging device unit, a light irradiation unit, an interface unit, and a camera control unit that controls these units. can be configured to include The imaging element section may include an imaging element such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), a filter, a microlens, and the like. The imaging device section may be a device capable of receiving light in the visible region to form a captured image, or may be a device capable of receiving light in the near-infrared region to form a captured image. The light irradiation section includes a light-emitting element such as an LED (Light Emitting Diode), and may include a near-infrared LED or the like so that the driver's face can be imaged day or night. The camera 11 captures an image at a predetermined frame rate (for example, several tens of frames per second), and data of the captured image is input to the image processing section 12 . The camera 11 may be an integrated type or an external type.

The image processing unit 12 is configured as an image processing device including one or more CPU (Central Processing Unit) 13 , ROM (Read Only Memory) 14 , and RAM (Random Access Memory) 15 . The ROM 14 includes a program storage unit 141 and a face feature amount storage unit 142 , and the RAM 15 includes an image memory 151 for storing an input image from the camera 11 . It should be noted that the driver monitoring device 10 may be provided with another storage unit, and the storage unit may be used as the program storage unit 141 , the facial feature amount storage unit 142 and the image memory 151 . The separate storage unit may be a semiconductor memory or a storage medium readable by a disk drive or the like.

The CPU 13 is an example of a hardware processor, and reads programs stored in the program storage unit 141 of the ROM 14 and data such as facial feature amounts stored in the facial feature amount storage unit 142, interprets and executes them. , the image input from the camera 11 is processed, for example, face image processing such as face detection processing is performed. In addition, the CPU 13 performs processing of outputting the result (for example, processing data, determination signal, control signal, etc.) obtained by the face image processing to the ECU 40 or the like via the communication unit 16 .

The face feature quantity storage unit 142 stores a specific individual face feature quantity 142a shown in FIG. A facial feature amount 142b is stored.
Various feature amounts effective for detecting a face from an image can be used as the learned face feature amount. For example, a feature amount (Haar-like feature amount) focused on the brightness difference in a local area of the face (difference in average luminance between two rectangular areas of various sizes) may be used. Alternatively, a feature value (LBP (Local Binary Pattern) feature value) focusing on a combination of luminance distributions in local regions of the face may be used, or a distribution of luminance gradient directions in local regions of the face may be used. A feature amount focused on a combination (HOG (Histogram of Oriented Gradients) feature amount) may be used.

The face feature quantity stored in the face feature quantity storage unit 142 is extracted as a feature quantity effective for face detection using various machine learning techniques, for example. Machine learning is a process of finding patterns inherent in data (learning data) by a computer. For example, AdaBoost may be used as an example of a statistical learning method. AdaBoost selects a large number of classifiers with low discrimination ability (weak classifiers), selects weak classifiers with a small error rate from among these many weak classifiers, adjusts parameters such as weights, and creates a hierarchical structure. It is a learning algorithm that can construct a strong discriminator by Note that the discriminator may also be called a discriminator, a classifier, or a learner.

The strong classifier is configured, for example, to classify one feature quantity effective for face detection by one weak classifier, select a large number of weak classifiers and their combinations by AdaBoost, and use these to perform hierarchical classification. It is good also as what constructed the structure. Note that one weak discriminator may output information such as 1 for a face and 0 for a non-face, for example. Also, as a learning method, a learning method called Real AdaBoost, which can output a real number between 0 and 1 instead of 0 or 1, may be used. Moreover, a neural network having an input layer, an intermediate layer, and an output layer may be used for these learning methods.

A large number of face images captured under various conditions and a large number of non-face images (non-face images) are given as learning data to a learning device equipped with such a learning algorithm. By adjusting parameters for optimization, it is possible to construct a strong discriminator having a hierarchical structure capable of detecting faces with high accuracy. One or more feature amounts used in the weak classifiers of each layer constituting such a strong classifier can be used as learned face feature amounts.

For example, the specific individual's face feature quantity 142a is obtained by individually capturing a specific individual's face image at a predetermined location in advance under various conditions (various conditions such as face orientation, line-of-sight direction, and eye open/closed state). Then, these many captured images are input to the learning device as training data, and adjusted by learning processing, and are parameters indicating the facial features of a specific individual. The face feature amount 142a of the specific individual may be, for example, a combination pattern of light and dark differences in local regions of the face obtained by learning processing. The face feature quantity 142a of the specific individual stored in the face feature quantity storage unit 142 may be the face feature quantity of only one specific individual, or may correspond to the case where a plurality of specific individuals drive the vehicle 2. , facial features of a plurality of specific individuals may be stored.

The normal face feature amount 142b is obtained by using images of normal people's faces captured under various conditions (various conditions such as face orientation, line of sight direction, and eye open/closed state) as teacher data. , and adjusted by the learning process, representing the features of a normal human face. The normal face feature quantity 142b may be, for example, a combination pattern of light and dark differences in local areas of the face obtained by learning processing. Information registered in a predetermined facial feature amount database may be used as the normal facial feature amount 142b.

The learned facial feature amounts stored in the facial feature amount storage unit 142 are stored in the facial feature amount storage unit 142 by, for example, fetching them from a server on the cloud via a communication network such as the Internet or a mobile phone network. It may be configured to be

The ECU 40 is composed of a computer device including one or more processors, memories, communication modules, and the like. A processor installed in the ECU 40 reads, interprets, and executes the program stored in the memory, thereby performing predetermined control on the actuator 42 and the like.

The ECU 40 includes, for example, at least one of a driving system ECU, a driving support system ECU, a body system ECU, and an information system ECU.

The travel system ECU includes, for example, a drive system ECU, a chassis system ECU, and the like. The drive system ECU includes, for example, a control unit related to a "running" function such as engine control, motor control, fuel cell control, EV (Electric Vehicle) control, or transmission control. The chassis system ECU includes, for example, a control unit for "stop, turn" functions such as brake control or steering control.

The driving support system ECU has, for example, an automatic braking support function, a lane keeping support function (also referred to as LKA/Lane Keep Assist), a constant speed driving/vehicle distance support function (also referred to as ACC/Adaptive Cruise Control), and a forward collision warning function. , Lane departure warning function, blind spot monitoring function, traffic sign recognition function, etc. Functions that automatically improve safety or realize comfortable driving in cooperation with driving system ECU etc. (driving support function or automatic driving function) may be configured to include at least one or more control units for

The driving support system ECU includes, for example, level 1 (driver support), level 2 (partially automated driving), and level 3 (conditional automated driving) in the automated driving levels presented by the Society of Automotive Engineers (SAE). ) may be equipped with at least one of the functions of Furthermore, the function of level 4 (highly automated driving) or level 5 (fully automated driving) of the automatic driving level may be equipped, or only the functions of levels 1 and 2 or only levels 2 and 3 are equipped. good too. Also, the in-vehicle system 1 may be configured as an automatic driving system.

The body system ECU may include at least one or more control units related to vehicle body functions such as door locks, smart keys, power windows, air conditioners, lights, meter panels, and blinkers.

The information system ECU may include, for example, an infotainment device, a telematics device, or an ITS (Intelligent Transport Systems) related device. The infotainment device may include, for example, an HMI (Human Machine Interface) device that functions as a user interface, a car navigation device, an audio device, and the like. The telematics device may include a communication unit or the like for communicating with the outside. The ITS-related device may include an ETC (Electronic Toll Collection System), a communication unit for performing road-to-vehicle communication with a roadside device such as an ITS spot, or vehicle-to-vehicle communication.

The sensor 41 may include various in-vehicle sensors that acquire sensing data necessary for the ECU 40 to control the operation of the actuator 42 . For example, in addition to vehicle speed sensors, shift position sensors, accelerator opening sensors, brake pedal sensors, steering sensors, etc., peripheral monitoring such as cameras for imaging outside the vehicle, radar such as millimeter waves, LIDER, and ultrasonic sensors Sensors and the like may also be included.

The actuator 42 is a device that executes operations related to running, steering, braking, etc. of the vehicle 2 based on control signals from the ECU 40, and includes, for example, an engine, a motor, a transmission, a hydraulic or electric cylinder, and the like.

[Example of functional configuration]
FIG. 3 is a block diagram showing a functional configuration example of the image processing section 12 of the driver monitoring device 10 according to the embodiment.
The image processing unit 12 includes an image input unit 21, a face detection unit 22, a specific individual determination unit 25, a first face image processing unit 26, a second face image processing unit 30, an output unit 34, and a face feature amount storage unit 142. is composed of

The image input unit 21 performs processing for capturing an image including the face of the driver 3 captured by the camera 11 .

The face detection unit 22 includes a specific individual face detection unit 23 and a normal face detection unit 24, and performs processing for detecting a face area while extracting a feature amount for detecting a face from an input image. conduct.
The specific individual's face detection unit 23 uses the specific individual's face feature amount 142 a read from the face feature amount storage unit 142 to perform processing for detecting a face area from the input image.
The normal face detection unit 24 uses the normal face feature amount 142b read from the face feature amount storage unit 142 to perform processing for detecting a face area from the input image.

A method for detecting a face region from an image is not particularly limited, but a method for detecting a face region at high speed and with high accuracy is adopted. For example, the face detection unit 22 extracts a feature amount for detecting a face in each search area while scanning a predetermined search area (search window) with respect to the input image. The face detection unit 22 may extract, for example, the difference in brightness (luminance difference) of local regions of the face, the edge strength, or the relationship between these local regions as feature quantities. Then, the face detection unit 22 uses the feature amount extracted from the search area, the normal face feature amount 142b read from the face feature amount storage unit 142, or the specific individual face feature amount 142a, to create a hierarchical structure ( A hierarchical structure that captures the details of the face from the hierarchy that captures the face roughly) judges whether it is a face or a non-face, and performs processing to detect a face area from the image.

The specific individual determination unit 25 uses the feature amount of the face area detected by the face detection unit 22 and the face feature amount 142a of the specific individual read from the face feature amount storage unit 142 to determine the face in the detected face area. is the face of a specific individual.

The specific individual determining unit 25 calculates a correlation coefficient as an index indicating the relationship between the feature quantity extracted from the face region and the face feature quantity 142a of the specific individual, for example, as an index indicating correlation, and calculates the calculated correlation coefficient. , it may be determined whether the face in the face area is the face of a specific individual. For example, the correlation of feature amounts such as Haar-like features (luminance differences) of one or more local areas within the face area may be obtained. If the correlation coefficient is greater than a predetermined threshold, the face in the detected face region is determined to be the face of a specific individual. may be determined not to be the face of

In addition, the specific individual determination unit 25 includes a learned learner that has performed machine learning using the specific individual's face feature amount 142a as a parameter in order to determine whether or not the face is that of a specific individual. A feature amount extracted from a face region is input to a learned learning device, and arithmetic processing of the learned learning device, for example, correlation analysis processing of the feature value is performed to determine whether or not the face is a specific individual face. The determination information may be acquired from the learning device. The learned learner may include a nonlinear discriminator or may include a linear discriminator. For example, the learned learner may include a support vector machine or a neural network.
Further, the specific individual determination unit 25 may determine whether or not the face in the detected face area is the face of a specific individual based on the determination result for one frame of the input image from the camera 11, It may be determined whether or not the face in the detected face area is the face of a specific individual, based on the determination results for a plurality of frames of the input image from the camera 11 .

The first face image processing unit 26 performs face image processing for a specific individual when the specific individual determining unit 25 determines that the face is that of a specific individual. The first face image processing unit 26 includes a specific individual's face orientation estimation unit 27, a specific individual's eye open/closed detection unit 28, and a specific individual's gaze direction estimation unit 29. A configuration for estimating or detecting facial behavior may also be included. Further, the first face image processing unit 26 may perform any one of face image processing for a specific individual using the face feature amount 142a of the specific individual. In addition, the facial feature amount storage unit 142 stores the learned feature amount that has undergone machine learning for performing face image processing for a specific individual, and uses the learned feature amount to perform facial image processing for the specific individual. Any one of the face image processing may be performed.

The specific individual's face direction estimating unit 27 performs processing for estimating the face direction of the specific individual. The specific individual's face direction estimation unit 27 detects, for example, the positions and shapes of facial features such as eyes, nose, mouth, and eyebrows from the face region detected by the specific individual's face detection unit 23, and detects the detected facial features. Based on the position and shape, the process of estimating the direction of the face is performed.

A method for detecting facial features from a facial region in an image is not particularly limited, but it is preferable to employ a method that can detect facial features at high speed and with high accuracy. For example, a method of creating a three-dimensional face shape model, fitting it to a face region on a two-dimensional image, and detecting the position and shape of each organ of the face can be adopted. As a technique for fitting a 3D face shape model to a human face in an image, for example, the technique described in Japanese Patent Application Laid-Open No. 2007-249280 can be applied, but it is not limited to this.

Further, the specific individual's face orientation estimation unit 27 uses, as estimation data of the specific individual's face orientation, for example, the pitch angle of vertical rotation (around the X axis), which is included in the parameters of the three-dimensional face shape model, A yaw angle for left-right rotation (around the Y-axis) and a roll angle for overall rotation (around the Z-axis) may be output.

The specific individual's eye open/close detection unit 28 performs processing for detecting the open/closed state of the eye of the specific individual. The specific individual's eye open/close detection unit 28, for example, based on the position and shape of the facial organs obtained by the specific individual's face direction estimation unit 27, particularly the position and shape of the eye feature points (eyelid, pupil), Detects the open/closed state, for example, whether the eyes are open or closed. The open/closed state of the eyes can be determined by, for example, using a learner in advance for the feature values of eye images (eyelid position, shape of pupil (black eye), size of white and black eye portions, etc.) in various eye open/closed states. It may also be detected by learning using and evaluating the degree of similarity with these learned feature amount data.

The specific individual's line-of-sight direction estimating unit 29 performs processing for estimating the direction of the specific individual's line of sight. The sight line direction estimating unit 29 of the specific individual, for example, based on the orientation of the face of the driver 3 and the position and shape of the facial organs of the driver 3, especially the position and shape of the characteristic points of the eyes (outer corners, inner corners, pupils), Estimate the direction of the line of sight. The line-of-sight direction is the direction in which the driver 3 is looking, and is determined by, for example, a combination of the direction of the face and the direction of the eyes.

In addition, the direction of the line of sight is, for example, the feature amount of the image of the eyes (the relative positions of the outer corner, the inner corner, and the pupil, or the relative positions of the white and black portions of the eye, shading, light and shade, etc.) in various combinations of face orientation and eye orientation. texture, etc.) may be learned in advance using a learning device, and the degree of similarity with the learned feature amount data may be evaluated for detection. In addition, the sight line direction estimation unit 29 of the specific individual estimates the size and center position of the eyeball from the size and orientation of the face, the position of the eyes, etc. using the fitting result of the three-dimensional face shape model, etc. , and a vector connecting the center of the eyeball and the center of the pupil may be detected as the line-of-sight direction.

When the specific individual determination unit 25 determines that the face is not that of a specific individual, the second face image processing unit 30 performs normal face image processing. The second face image processing unit 30 includes a normal face orientation estimation unit 31, a normal eye open/close detection unit 32, and a normal gaze direction estimation unit 33. It may also include an estimating or detecting configuration. Further, the second face image processing section 30 may perform any of normal face image processing using the normal face feature amount 142b. In addition, a learned feature amount obtained by performing machine learning for performing normal face image processing is stored in the face feature amount storage unit 142, and normal face image processing is performed using the learned feature amount. Either processing may be performed. The processing performed by the normal face orientation estimation unit 31, the normal eye open/closed detection unit 32, and the normal gaze direction estimation unit 33 is performed by the specific individual's face orientation estimation unit 27 and the specific individual's eye open/closed detection. Since the unit 28 is basically the same as the specific individual's line-of-sight direction estimation unit 29, the description thereof will be omitted here.

The output unit 34 performs processing for outputting information based on image processing by the image processing unit 12 to the ECU 40 or the like. The information based on the image processing may be, for example, information related to facial behavior such as the direction of the face of the driver 3, the direction of the line of sight, or the state of opening and closing the eyes, or the driver 3 determined based on the detection result of the facial behavior. (for example, states such as looking ahead, looking aside, dozing off, looking backward, lying down, etc.). Information based on image processing may be a predetermined control signal (a control signal for performing attention or warning processing, or a control signal for controlling the operation of the vehicle 2, etc.) based on the state determination of the driver 3.

[Processing operation example]
FIG. 4 is a flowchart showing an example of processing operations performed by the CPU 13 of the image processing unit 12 in the driver monitoring device 10 according to the embodiment. The camera 11 captures, for example, several tens of frames per second, and this process is performed for each frame or for each frame at regular intervals.

First, in step S1, the CPU 13 operates as the image input unit 21, reads an image captured by the camera 11 (an image including the face of the driver 3), and proceeds to step S2.

In step S2, the CPU 13 operates as a normal face detection unit 24, performs normal face detection processing on the input image, and proceeds to step S3. For example, the CPU 13 extracts a feature amount for detecting a face in each search area while scanning a predetermined search area (search window) with respect to the input image. Then, the CPU 13 uses the feature amount extracted from the search area and the normal face feature amount 142b read from the face feature amount storage unit 142 to determine whether it is a face or a non-face, and detects a face area from the image. process.

In step S3, the CPU 13 operates as the specific individual's face detection unit 23, performs specific individual's face detection processing on the input image, and proceeds to step S4. For example, the CPU 13 extracts a feature amount for detecting a face in each search area while scanning a predetermined search area (search window) with respect to the input image. Then, the CPU 13 uses the feature amount extracted from the search area and the face feature amount 142a of the specific individual read from the face feature amount storage unit 142 to determine whether it is a face or a non-face, and extracts the face area from the image. Perform detection processing. The processes of steps S2 and S3 may be performed in parallel within one step, or may be performed in combination.

In step S4, the CPU 13 operates as the specific individual determining unit 25, and uses the feature amount of the face area detected in steps S2 and S3 and the facial feature amount 142a of the specific individual read from the facial feature amount storage unit 142. Then, a process of determining whether or not the face in the face area is the face of a specific individual is performed, and the process proceeds to step S5.

In step S5, the CPU 13 determines whether or not the result of the determination processing in step S4 is the face of a specific individual.

In step S6, the CPU 13 operates as the specific individual's face orientation estimating unit 27, for example, detects the position and shape of facial organs such as eyes, nose, mouth, and eyebrows from the face area detected in step S3. The orientation of the face is estimated based on the position and shape of the facial features, and the process proceeds to step S7.

In step S7, the CPU 13 operates as the eye open/close detection unit 28 of the specific individual, and, for example, based on the position and shape of the facial features obtained in step S6, particularly the position and shape of the characteristic points of the eyes (eyelids, pupils). , the open/closed state of the eyes, for example, whether the eyes are open or closed, is detected, and the process proceeds to step S8.

In step S8, the CPU 13 operates as the gaze direction estimating unit 29 of the specific individual, for example, the direction of the face obtained in step S6, the position and shape of the facial organs, especially the characteristic points of the eyes (outer corners, inner corners, pupils). Based on the position and shape, the direction of the line of sight is estimated, and then the process ends.

On the other hand, if the CPU 13 determines in step S5 that the face is not that of a specific individual, in other words, that it is a normal face, the process proceeds to step S9.
In step S9, the CPU 13 operates as a normal face direction estimating unit 31, for example, detects the positions and shapes of facial organs such as eyes, nose, mouth, and eyebrows from the face area detected in step S2, and detects the detected face. The orientation of the face is estimated based on the position and shape of the organ, and the process proceeds to step S10.

In step S10, the CPU 13 operates as a normal eye open/close detection unit 32. For example, based on the position and shape of the facial features obtained in step S9, particularly the position and shape of the characteristic points of the eyes (eyelids, pupils), The open/closed state of the eyes, for example, whether the eyes are open or closed is detected, and the process proceeds to step S11.

In step S11, the CPU 13 operates as a normal gaze direction estimator 33. For example, the direction of the face obtained in step S9, the positions and shapes of facial organs, especially the positions of characteristic points of the eyes (outer corners, inner corners, pupils) of the eyes. Based on the shape and shape, the direction of the line of sight is estimated, and then the process ends.

FIG. 5 is a flowchart showing an example of specific individual determination processing operation performed by the CPU 13 of the image processing unit 12 in the driver monitoring device 10 according to the embodiment. This processing operation is an example of the specific individual determination processing operation in step S4 shown in FIG. 4, and is an example of the processing operation in the case of determination with one input image (one frame).

First, in step S21, the CPU 13 reads the feature amount extracted from the face area detected by the face detection processing in steps S2 and S3 shown in FIG. The specific individual's face feature quantity 142a that has already been read is read, and the process proceeds to step S23.

In step S23, the CPU 13 performs a process of calculating a correlation coefficient between the feature amount extracted from the face area read in step S21 and the face feature amount 142a of the specific individual read in step S22, and proceeds to step S24. proceed.

In step S24, the CPU 13 determines whether the calculated correlation coefficient is greater than a predetermined threshold value for determining whether or not the individual is a specific individual. If it is determined that the feature quantity extracted from the face region and the face feature quantity 142a of the specific individual are highly correlated (in other words, the similarity is high), the process proceeds to step S25.
In step S25, the CPU 13 determines that the face detected in the face area is the face of a specific individual, and then terminates the process.

On the other hand, in step S24, the correlation coefficient is equal to or less than a predetermined threshold value, in other words, the correlation between the feature quantity extracted from the face region and the face feature quantity 142a of the specific individual is low (in other words, the similarity is low), the process proceeds to step S26.
In step S26, the CPU 13 determines that the face is not that of a specific individual, in other words, that it is a normal face, and then terminates the process.

FIG. 6 is a flowchart showing an example of specific individual determination processing operation performed by the CPU 13 of the image processing unit 12 in the driver monitoring device 10 according to the embodiment.
This processing operation is another example of the specific individual determination processing operation in step S4 shown in FIG. 4, and is an example of the processing operation in the case of determination using a plurality of input images (a plurality of frames).

First, in step S31, the CPU 13 sets the counter (numSp) of the image in which the face of the specific individual is detected to 0, sets the counter (i) of the input image to be judged to 0 in step S32, and sets the counter (i) of the input image to be determined to 0 in step S33. proceed to

In step S33, the CPU 13 performs specific individual determination processing (for example, the processing of steps S21 to S26 shown in FIG. 5) for one input image (one frame), and proceeds to step S34.

In step S34, the CPU 13 determines whether or not the result of the determination processing in step S33 is the face of a specific individual.

In step S35, the CPU 13 adds 1 to the counter (numSp) of the face image of the specific individual, and advances the process to step S36. The process proceeds to S37.

On the other hand, if the CPU 13 determines in step S34 that the face is not that of a specific individual, the process proceeds to step S36. In step S36, 1 is added to the counter (i) of the input image, and the process proceeds to step S37. . That is, in this case, 1 is not added to the face image counter (numSp) of the specific individual.

In step S37, the CPU 13 determines whether or not the input image counter (i) is less than a predetermined number of images N. If the counter (i) is less than N (in other words, the predetermined number of images (N If it is determined that the determination of the image has not been completed, the process returns to step S33, and the process of determining the specific individual in the next input image is repeated.

On the other hand, in step S37, if the CPU 13 determines that the counter (i) of the input image is not less than N (in other words, the predetermined number of images (N) has been determined), the process proceeds to step S38. proceed.

In step S38, the CPU 13 determines whether or not the face image counter (numSp) of the specific individual is greater than a predetermined threshold for determining that the individual is a specific individual. For example, when the number of input images used for determination is N (N frames), the predetermined threshold may be set to N/2, or may be set to a value larger than N/2. .

In step S38, if the CPU 13 determines that the counter (numSp) of the face image of the specific individual is larger than the predetermined threshold value, the process proceeds to step S39. After finishing the determination processing for the specific individual, face image processing for the specific individual is performed.

On the other hand, if the CPU 13 determines in step S38 that the counter (numSp) of the specific individual's face image is equal to or less than the predetermined threshold value, the process proceeds to step S40. For example, it is a normal face), the specific individual determination process is completed, and then normal face image processing is performed.

According to the driver monitoring device 10 according to the above-described embodiment, the facial feature amount storage unit 142 stores the facial feature amount 142a of the specific individual and the normal facial feature amount 142b as the learned facial feature amount, The specific individual determination unit 25 determines whether or not the face in the face region is the face of the specific individual by using the feature amount of the face region detected by the face detection unit 22 and the face feature amount 142a of the specific individual. be done. Therefore, by using the face feature amount 142a of a specific individual, it is possible to accurately determine whether or not it is the face of a specific individual, and reduce the load on the determination process.

Further, when the specific individual determination unit 25 determines that the face is that of a specific individual, the first face image processing unit 26 can perform the specific individual's face image processing with high accuracy. On the other hand, when the specific individual determining unit 25 determines that the face is not that of a specific individual, in other words, that it is a normal face (face of a person other than the specific individual), the second face image processing unit 30 determines that the face is a normal face. Image processing can be performed with high accuracy. Therefore, regardless of whether the driver 3 is a specific individual or a normal person other than the specific individual, it is possible to accurately sense each face.

Further, the specific individual determination unit 25 calculates a correlation coefficient as an index indicating the correlation between the feature amount extracted from the face region and the specific individual's facial feature amount 142a, and based on the calculated correlation coefficient, It is determined whether or not the face in the face area is the face of the specific individual. Thereby, it is possible to efficiently determine whether or not the face in the face region is the face of the specific individual based on the correlation coefficient. Thus, the processing efficiency of the determination can be further improved.

The in-vehicle system 1 also includes a driver monitoring device 10 and one or more ECUs 40 that execute predetermined processing based on the monitoring results output from the driver monitoring device 10 . Therefore, it is possible to cause the ECU 40 to appropriately perform predetermined control based on the monitoring result. As a result, it is possible to construct a highly safe vehicle-mounted system that allows even specific individuals to drive with peace of mind.

Although the embodiments of the present invention have been described in detail above, the above description is merely an example of the present invention in every respect. It goes without saying that various modifications and changes can be made without departing from the scope of the invention.
In the above embodiment, the case where the image processing device according to the present invention is applied to the driver monitoring device 10 has been described, but application examples are not limited to this. For example, in a device or system that monitors people who operate or monitor various facilities such as machines and devices in a factory, or perform predetermined work, the above-mentioned specific individuals are included in the monitoring targets. In this case, the image processing apparatus according to the present invention can be applied.

[Appendix]
Embodiments of the present invention can also be described in the following appendices, but are not limited thereto.
(Appendix 1)
An image processing device (12) for processing an image input from an imaging unit (11),
A facial feature amount storage unit that stores a specific individual's facial feature amount (142a) and a normal facial feature amount (142b) as learned facial feature amounts that have been trained to detect a face from the image. (142) and
a face detection unit (22) for detecting a face region while extracting a feature amount for detecting a face from the image;
A specific individual determination unit that determines whether or not the face in the face area is the face of the specific individual by using the feature amount of the detected face area and the face feature amount of the specific individual (142a). (25) and
a first face image processing unit (26) that performs face image processing for a specific individual when the specific individual determining unit (25) determines that the face is that of the specific individual;
and a second face image processing unit (30) for performing normal face image processing when the specific individual determining unit (25) determines that the face is not that of the specific individual. Device.

(Appendix 2)
an imaging unit (11), a storage unit (14),
using a device comprising at least one or more processors (13),
An image processing method for processing an image input from the imaging unit (11),
The storage unit (14) is
A facial feature amount storage unit that stores a specific individual's facial feature amount (142a) and a normal facial feature amount (142b) as learned facial feature amounts that have been trained to detect a face from the image. (142),
the processor (13)
a face detection step (S2, S3) of detecting a face region while extracting a face feature amount from the image;
Using the feature amount of the face area detected by the face detection step (S2, S3) and the face feature amount (142a) of the specific individual read from the face feature amount storage unit (142), the a specific individual determination step (S4) for determining whether or not the face in the face area is the face of the specific individual;
a first face image processing step (S6, S7, S8) for performing face image processing for a specific individual when the specific individual determining step (S4) determines that the face is that of the specific individual;
and a second face image processing step (S9, S10, S11) of performing normal face image processing when it is determined by the specific individual determination step (S4) that the face is not that of the specific individual. Image processing method.

1 in-vehicle system 2 vehicle 3 driver 10 driver monitoring device 11 camera 12 image processing unit 13 CPU
14 ROMs
141 program storage unit 142 facial feature amount storage unit 142a specific individual facial feature amount 142b normal facial feature amount 15 RAM
151 image memory 16 communication unit 21 image input unit 22 face detection unit 23 specific individual face detection unit 24 normal face detection unit 25 specific individual determination unit 26 first face image processing unit 27 specific individual face direction estimation unit 28 specific individual Eye open/close detection unit 29 Specific individual gaze direction estimation unit 30 Second face image processing unit 31 Normal face direction estimation unit 32 Normal eye open/close detection unit 33 Normal gaze direction estimation unit 34 Output unit 40 ECU
41 sensor 42 actuator 43 communication bus

Claims (11)

An image processing device that processes an image input from an imaging unit,
As the learned face feature quantity that has been trained to detect a face from the image, the face feature quantity of a specific individual having face features different from the face features of a general person , and the face feature quantity of a specific individual other than the specific individual a facial feature amount storage unit that stores normal facial feature amounts used when the person is a person ;
a face detection unit that detects a face area while extracting a feature amount for detecting a face from the image;
a specific individual determination unit that determines whether or not the face in the face region is the face of the specific individual using the detected feature quantity of the face region and the face feature quantity of the specific individual;
a first face image processing unit that performs face image processing for a specific individual when the specific individual determining unit determines that the face is the face of the specific individual;
A second face image processing unit that performs normal face image processing used when it is a general person other than the specific individual when it is determined by the specific individual determination unit that the face is not the specific individual ,
The image processing apparatus, wherein the specific individual includes a person whose facial features are partially missing or deformed, or a person whose facial features are displaced from an average position. The specific individual determination unit
calculating an index indicating a correlation between the feature quantity extracted from the face region and the face feature quantity of the specific individual;
2. The image processing apparatus according to claim 1, wherein whether or not the face in the face area is the face of the specific individual is determined based on the calculated index. The specific individual determination unit
determining that the face in the face region is the face of the specific individual when the index is greater than a predetermined threshold;
3. The image processing apparatus according to claim 2, wherein when said index is equal to or less than said predetermined threshold value, it is determined that the face in said face area is not the face of said specific individual. The specific individual determination unit
4. The method according to any one of claims 1 to 3, wherein it is determined whether or not the face in the face area is the face of the specific individual based on the determination result for one frame of the image. The image processing device according to . The specific individual determination unit
4. The method according to any one of claims 1 to 3, wherein it is determined whether or not the face in the face area is the face of the specific individual based on determination results for a plurality of frames of the image. The image processing device according to . 6. The face image processing includes at least one of face detection processing, face direction estimation processing, line-of-sight direction estimation processing, and eye open/close detection processing. The image processing device according to the item. The image processing device according to any one of claims 1 to 6,
an imaging unit that captures an image to be input to the image processing device;
and an output unit for outputting information based on image processing by the image processing device. a monitoring device according to claim 7;
A control system comprising one or more control devices communicably connected to the monitoring device and executing predetermined processing based on the information output from the monitoring device. the monitoring device is a device for monitoring a driver of a vehicle;
9. The control system of claim 8, wherein said controller includes an electronic control unit mounted on said vehicle. An image processing method for processing an image input from an imaging unit,
a face detection step of detecting a face region while extracting face feature amounts from the image;
The learned specific individual who has undergone learning to detect the face of a specific individual having facial features different from the facial features of a general person, and the feature amount of the facial region detected by the face detection step. a specific individual determination step of determining whether or not the face in the face region is the face of the specific individual using the facial feature amount of
a first face image processing step of performing face image processing for a specific individual when the face is determined to be that of the specific individual by the specific individual determination step;
and a second face image processing step of performing normal face image processing used in the case of a general person other than the specific individual when it is determined by the specific individual determination step that the face is not that of the specific individual. ,
An image processing method, wherein the specific individual includes a person whose facial organs are partially missing or deformed, or a person whose facial organs are displaced from an average position. A program for causing at least one or more computers to process an image input from an imaging unit,
to said at least one computer;
a face detection step of detecting a face region while extracting face feature amounts from the image;
The learned specific individual who has undergone learning to detect the face of a specific individual having facial features different from the facial features of a general person, and the feature amount of the facial region detected by the face detection step. a specific individual determination step of determining whether or not the face in the face region is the face of the specific individual using the facial feature amount of
a first face image processing step of performing face image processing for a specific individual when the face is determined to be that of the specific individual by the specific individual determination step;
and a second face image processing step of performing normal face image processing used in the case of a general person other than the specific individual when it is determined by the specific individual determining step that the face is not that of the specific individual. ,
A program, wherein the specific individual includes a person whose facial organs are partially missing or deformed, or a person whose facial organs are displaced from an average position.

Images