JP5289993B2 - TRACKING DEVICE AND TRACKING METHOD - Google Patents

Description

  The present invention relates to a tracking device and a tracking method for sequentially processing consecutive images and tracking a face appearing in each image.

  Conventionally, a face detection technique for detecting a face of a person or the like shown in an image is known. Such a technique detects the position of a face in an image by, for example, a template matching method, and detects the position of a facial feature point such as an eye, nose, or mouth, and can detect the size and orientation of the face. it can. For example, in an imaging apparatus such as a digital camera, an object image formed on an imaging element is displayed in real time (live view) and used as an electronic viewfinder. However, in recent years, a live view image is applied by applying face detection technology. A device that detects the inside face and displays a face frame indicating the detected face is known. The face detection result is also used for exposure and focus control. For example, Patent Literature 1 discloses a technique for performing focus detection based on the result of face detection (face recognition).

  On the other hand, there is known a motion detection technique for detecting a motion between images by performing pattern matching between successive images and calculating a movement amount thereof.

JP 2006-227080 A

  In the conventional face detection technology, for example, when the face is facing sideways or behind, the detection accuracy may be lowered. For this reason, for example, if the orientation of the face changes, the face detection result may become unstable. When the above-mentioned face frame is displayed on the live view image according to the face detection result, the display / non-display of the face frame is repeated. Sometimes it was difficult to see due to flickering.

  On the other hand, if motion detection is performed on a face detected by face detection, the face motion can be detected even when face detection fails due to a significant change in the orientation of the face. You can track without losing your face. However, when a plurality of faces are detected from an image, there is a problem that processing load increases when motion detection is performed for all the faces. This problem is particularly problematic when applied to an apparatus having a limited processing capability such as a digital camera. In addition, if time is required for face detection and motion detection, there is a problem that detection is not possible in a moving scene, and there is a problem of missing a photographing opportunity, and stable face tracking cannot be performed.

  The present invention has been made in view of the above, and provides a tracking device and a tracking method capable of stably tracking a face appearing in a continuous image without increasing the processing load. With the goal.

  In order to solve the above-described problems and achieve the object, a tracking device according to the present invention is a tracking device that tracks a face that appears in a continuous image, and sequentially processes the continuous image to include the image in the image. A face detection section for detecting a plurality of faces, a face area setting section for setting a face area including each face detected by the face detection section for each face, and a face detection result by the face detection section. Based on the difficulty level evaluation unit that evaluates the detection difficulty level of each detected face and a face area that satisfies the predetermined condition of the detection difficulty level of each face from the face areas including each face And a motion detection target that is set by switching whether or not to make a motion detection target for each of the face regions including each face based on the selection result by the face region selection unit The motion detection target between the setting unit and adjacent images Characterized in that it comprises a motion detector for detecting a movement of the target region set by the tough, the.

  In the tracking device according to the present invention, in the above invention, the face area selection unit selects a face area including a predetermined number of faces with high detection difficulty, and the motion detection target setting unit includes the face A face area including a predetermined number of faces selected by the area selection unit is set as the target area.

  In the tracking device according to the present invention, in the above invention, the face area selection unit selects a face area including a predetermined number of faces with a low detection difficulty level, and the motion detection target setting unit includes the face A face area including a predetermined number of faces selected by the area selection unit is not set as the target area.

  In the tracking device according to the present invention as set forth in the invention described above, the face detection unit outputs at least one of a face size, a face position, a face orientation, and a face tilt as the face detection result. The difficulty level evaluation unit, based on the face detection result of each face detected by the face detection unit, at least a face size, a face position, a face orientation, a face tilt, a face orientation change The degree of difficulty in detecting each face is evaluated using one or more of a change in face inclination, a face moving speed, and a face moving direction as evaluation parameters.

  In the tracking device according to the present invention, in the above invention, the difficulty level evaluation unit estimates the possibility that each detected face overlaps with another face, and uses the estimation result as the evaluation parameter for each face. It is characterized by evaluating the degree of detection difficulty.

  In the tracking device according to the present invention as set forth in the invention described above, the difficulty level evaluation unit determines the moving speed and / or moving direction of the face from the face detection result of each face, and based on the determination result. Further, the possibility that each face overlaps with another face is estimated.

  In the tracking device according to the present invention, in the above invention, the difficulty level evaluation unit includes a weighting unit that weights the evaluation parameter using a weighting factor set in advance for each evaluation parameter. The degree of detection difficulty of each face is evaluated based on the evaluation parameter weighted by the unit.

  Further, the tracking device according to the present invention is based on the face area including each face detected by the face detection unit and the movement of the target area detected by the motion detection unit in the above invention. The image processing apparatus includes a face area determination unit that determines a face area in the image.

  Further, the tracking device according to the present invention includes a display processing unit that switches the continuous images and performs display processing on the display unit in the above invention, and the display processing unit includes the face area in the determined image. According to the above, a face frame indicating the face in the image is displayed.

  In the tracking device according to the present invention, in the above-described invention, an imaging unit that captures a subject for each frame and sequentially generates the continuous image, a shooting instruction unit that performs a shooting instruction, and the determined image An imaging condition setting unit configured to set an imaging condition of the imaging unit using a latest face detection result detected by the face detection unit for a face in an inside face region.

  The tracking method according to the present invention is a tracking method for tracking a face that appears in a continuous image, and a face detection step for detecting a plurality of faces in the image by sequentially processing the continuous image; A face area setting step for setting a face area including each face detected in the face detection step for each face, and each detected face based on a face detection result in the face detection step A difficulty level evaluation step for evaluating the detection difficulty level, a face area selection step for selecting a face area where the detection difficulty level of each face satisfies a predetermined condition from the face areas including the faces, and the face area Based on the selection result in the selection step, between the motion detection target setting step of switching and setting whether or not to make a motion detection target region for each face region including each face, and between adjacent images , The target region set in the motion detection target setting step Characterized in that it comprises a motion detector for detecting gas, a.

  According to the present invention, for each face area including each face detected by face detection, whether or not to make a target area for motion detection according to the detection difficulty level of each face evaluated based on the face detection result. Can be set by switching. Then, the motion detection of the face area as the target area can be performed. Therefore, there is an effect that a face appearing in a continuous image can be tracked stably without increasing the processing load.

  DESCRIPTION OF EMBODIMENTS Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. In this embodiment, a case where the tracking device of the present invention is applied to a digital camera will be described as an example. Note that the present invention is not limited to the embodiments. Moreover, in description of each drawing, the same code | symbol is attached | subjected and shown to the same part.

(Embodiment)
FIG. 1 is a rear view of the digital camera 1. As shown in FIG. 1, the digital camera 1 includes a release switch (shutter switch) 3 for instructing photographing timing provided on the upper surface of the camera body 2 and a power switch 4 provided on the back surface of the camera body 2. And a menu switch 5, a cross key 6 having up / down / left / right direction switches (up switch, down switch, left switch and right switch), an OK switch 7 for confirming the operation content, various shooting modes and playback modes, etc. A mode dial 8 for switching modes, a display unit 24 for displaying various screens, and the like are provided. The release switch 3 is, for example, a two-stage press button. When the release switch 3 is half-pressed, the first release switch is turned ON, and when the release switch 3 is fully pressed, the second release switch is turned ON. Although not shown, a flash, an imaging lens, and the like are disposed on the front surface of the camera body 2.

  When the power switch 4 is pressed and the power is turned on in the digital camera 1, the digital camera 1 is ready for photographing if the photographing mode is selected with the mode dial 8. In the shooting mode, a subject image incident through the imaging lens is output every frame (for example, 1/30 second) and is displayed in real time on the display unit 24 as a live view image. While viewing the image, the release switch 3 is pressed to shoot a still image or a moving image. On the other hand, if the playback mode is selected with the mode dial 8 when the power is turned on, the digital camera 1 enters the playback mode. In this playback mode, the user enjoys displaying (playing back) still images and moving images captured by the digital camera 1 on the display unit 24.

  First, an outline of a face detection function that is one of the functions of the digital camera 1 of the present embodiment will be described. The digital camera 1 according to the present embodiment includes a face detection unit 17 (see FIG. 8) that detects the face of a person or the like shown in the image, and performs image processing on the live view image captured every frame. Perform detection. And the process which displays the face frame which shows the detected face on a live view image is performed.

  FIG. 2 is a diagram showing an example of four live view images updated and displayed on the display unit 24 in time series. As described above, the face detection technique detects the position of the face in the image by the template matching method, and detects the position of the feature point of the face such as the eyes, nose and mouth. The detection accuracy may decrease when the camera is facing or when the face is greatly inclined. For example, in the frame I11 shown in FIG. 2A, the face detection is successful because the face of the person P shown in the live view image is facing the front, and the face frame N11 is displayed. On the other hand, in the frame I13 in which the person P moves as shown in FIG. 2B, the face is facing sideways and the angle is greatly inclined, so the face detection fails and no face frame is displayed. In the subsequent frame I15 in FIG. 2C, the face of the person P cannot be detected and the face frame is not displayed. In the frame I17 shown in FIG. 2 (d), the face detection of the person P is successful and the face frame N14 is displayed.

  As described above, when the face direction or angle changes between frames, the display / non-display of the face frame may be repeated. In particular, such a situation occurs when a person moves violently, and the face frame flickers and is difficult to see. The face detection result is used for exposure and focus control, but there is a problem that these controls are not stable. In this embodiment, motion detection of a face area (face area) is performed in conjunction with face detection in a live view image, and even if face detection fails, tracking can be performed without losing sight of the face. It is.

  FIG. 3 is a diagram schematically showing an example of three live view images showing the faces F1 to F5 of five persons along a time series. In FIG. 3, in each frame I21, I23, and I25, FIG. Each face detected by face detection is shown surrounded by a one-dot chain line. Here, the frames I21, I23, and I25 show how the orientation and inclination of each face reflected in the live view image change in time series, and show the live view images of successive frames. is not. For example, in the frame I21 in FIG. 3A, all five faces F1 to F5 are detected by face detection. On the other hand, in the frame I23 in FIG. 3B, the face detection has failed because the face is greatly inclined with respect to the upper left face F1. In addition, in the frame I25 in FIG. 3C, in addition to the face detection failure for the face F1 as in FIG. 3B, the faces also appear on the lower left and lower right faces F2 and F5. Face detection failed due to facing backwards.

  FIG. 4 is a diagram showing an example of three live view images similar to FIG. 3. In FIG. 4, the faces detected by face detection in each of the frames I21, I23, and I25 are surrounded by a one-dot chain line. In addition, the face areas detected by motion detection in each of the frames I21, I23, and I25 are surrounded by a two-dot chain line. Although details will be described later, in motion detection, pattern matching is performed between adjacent frames using a face area (face area) including a face detected by face detection as a motion detection area (target area), and the amount of movement is calculated. If a motion detection area is associated between frames, tracking can be performed without losing sight of the face regardless of the orientation and inclination of the face. For example, in the example shown in FIG. 4, all five faces can be detected in each of the frames I21 to I25 shown in (a) to (c).

  Here, in face detection using the template matching method, a plurality of faces (for example, several tens) appearing in one image can be detected simultaneously. On the other hand, when motion detection is performed for a plurality of face areas, each face area must be set as a motion detection area, and pattern matching must be performed individually. If these are performed simultaneously, the processing load increases. For this reason, in order to ensure the necessary processing speed, for example, when motion detection is realized by hardware, there is a problem that the circuit scale increases.

  Therefore, in this embodiment, the number of motion detection areas in which motion detection can be simultaneously performed is determined in advance. Then, the priority of the face is scored based on the face detection result, and when the number of faces larger than the number of motion detection areas appears in the live view image, the motion detection areas are ordered in order from the face with the highest score. A number of face areas are selected as motion detection areas. 5 to 7 are diagrams for explaining the principle of setting the motion detection area. In each figure, the five persons shown in FIGS. 3 (a) to (c) and FIGS. 4 (a) to (c), respectively. Three live view images showing human faces F1 to F5 are shown. Although the number of motion detection areas is described as “3” here, the number of motion detection areas can be determined as appropriate according to the actual processing capability of the digital camera 1.

  For example, when faces F1 to F5 are detected by face detection as shown by being surrounded by a one-dot chain line in FIG. 5, the priority is scored for each of the faces F1 to F5. Although details of scoring priority will be described later, scoring is performed so that the priority becomes higher as it is assumed that the detection difficulty of face detection performed in the next frame is higher based on the face detection result for the face. I do. That is, in the present embodiment, the priority of scoring is evaluated in this way, thereby evaluating the detection difficulty level of each face detected by face detection. For example, as a result of face detection, if the face orientation is off the front, the face is tilted, or the face orientation or angle changes significantly compared to the previous frame, the next frame Therefore, it is considered that there is a high possibility that the face detection of the face will fail. Then, a face area as a motion detection area is selected based on the scored priority.

  For example, as a result of scoring the priorities for the faces F1 to F5 shown in FIG. 5, when the priorities are scoring high for the three faces F1, F4, and F5 surrounded by solid lines, The face areas of the faces F1, F4, and F5 are set as motion detection areas. Then, for each motion detection area, for example, by performing motion detection with the next frame, these faces F1, F4, and F5 are tracked.

  As a result of performing face detection in each frame in this way and performing motion detection on the face area as the motion detection area, for example, in frame I23, as shown by being surrounded by a two-dot chain line in FIG. Face areas of the faces F1, F4, and F5 are detected and tracked by motion detection, respectively. On the other hand, the face detection succeeds for the faces F2 to F5 and fails for the face F1. As a result, it is possible to continuously display the face frame for the face F1 whose face detection has failed in the frame I23 based on the motion detection result.

  In the present embodiment, for example, a face area that is used as a motion detection area in motion detection performed with the next frame is selected each time. At this time, for example, a face area that could be detected by motion detection but failed to detect the face, such as the face F1 in FIG. 6A, is always selected as the motion detection area. If the number of face areas selected as the motion detection area is less than the number of motion detection areas, a face area with a high priority scored for each face is further selected based on the face detection result.

  For example, in the example of FIG. 6A, there is one face area (face F1) that could be detected by motion detection but failed to detect the face, and the number of motion detection areas is less than “3”. Motion detection can be performed for the face areas of two of the successful faces F2 to F5. For example, it is assumed that the priority is scored in the order of the faces F5, F2, F4, and F3 as a result of scoring the priorities for the faces F2 to F5 based on the face detection results performed for the frame I23. Here, the priority of the face selected as the previous motion detection area may be scored low. For example, in the priority scoring results in the order of the faces F5, F2, F4, F3 illustrated, the priority of the face F2 is scored higher than the face F4 selected as the previous motion detection area. This is because in the frame I23, the face F2 has changed significantly compared to the face F4 from the previous frame. In this case, the face area of each of the faces F1, F2, and F5 indicated by a solid line in FIG. 6B is set as a motion detection area, and motion detection is performed between each motion detection area and the next frame. Thus, each of these faces F1, F2, and F5 is tracked.

  As a result, for example, as indicated by being surrounded by a two-dot chain line in FIG. 7, the face areas of the faces F1, F2, and F5 are detected and tracked by the motion detection in the frame I25. On the other hand, the face detection succeeds for the faces F3 and F4 and fails for the faces F1, F2, and F5. According to this, it is possible to continuously display the face frame for each of the faces F1, F2, and F5 whose face detection has failed in the frame I25, based on the motion detection result. In this way, it is possible to appropriately set the motion detection area by scoring the priority of each face detected by face detection for each frame and setting the face area of the face with high priority as the motion detection area. it can. Therefore, it is possible to realize stable tracking even when a face that is difficult to detect a face appears while limiting the number of motion detection areas to a predetermined number (number of motion detection areas) and suppressing an increase in processing load within an allowable range. .

  Next, the configuration of the digital camera 1 will be described. FIG. 8 is a schematic block diagram illustrating a configuration example of the digital camera 1. As shown in FIG. 8, the digital camera 1 includes an imaging optical system 11, an imaging device 12, an AFE (Analog Front End) 13, a frame memory 14, a motion detection unit 15, an image processing unit 16, a face detection unit 17, a face area. A face area selection unit 18 as a selection unit and a motion detection target setting unit, a recording medium I / F 19, a recording medium holding unit 20, a recording medium 21, a video encoder 22, a display driver 23, a display unit 24, a video signal output terminal 25, An operation unit 26, a RAM 27, a ROM 28, a display processing unit, a controller 29 as an imaging condition setting unit, and the like are provided.

  The imaging optical system 11 includes an imaging lens, a diaphragm, a shutter, and the like, and forms an incident subject image on the imaging element 12. The image pickup device 12 is a solid-state image pickup device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor), for example, and receives a light beam from a subject via the image pickup optical system 11 and photoelectrically converts it into frame units. Image data (analog electrical signal) is obtained. The AFE 13 performs analog signal processing such as CDS (Correlated Double Sampling) and AGC (Automatic Gain Control) on the image data obtained by the image sensor 12, and then performs A / D conversion processing to convert it into a digital electrical signal. To do. The image data digitized by the AFE 13 is output to the frame memory 14 and the motion detection unit 15 and is temporarily recorded in the RAM 27.

  The frame memory 14 is used as a working memory by the motion detection unit 15. This frame memory 14 has an area for storing image data for two frames, and at the time of displaying a live view image, the image data of the live view image (current frame image) of the current frame and the data captured immediately before are displayed. Image data of a live view image (previous frame image) of the immediately preceding frame is recorded.

  The motion detection unit 15 is for detecting the motion between frames based on the image data from the AFE 13. For example, the motion detection unit 15 moves between images captured from the image sensor 12 and output from the AFE 13 for each frame. The motion of the motion detection area between frames is detected by obtaining the motion vector of the detection area. Specifically, the motion detection unit 15 sets the face area selected by the face area selection unit 18 as a motion detection area. Then, the motion detection unit 15 performs pattern matching between the immediately preceding frame image input from the AFE 13 and recorded in the frame memory 14 and the current frame image, and a motion vector representing the amount of movement for each motion detection area. Is calculated.

  FIG. 9 is a diagram illustrating an example of the motion detection area set in the immediately preceding frame image, and the face area selected by the face area selecting unit 18 with respect to the angle of view range Ev of the immediately preceding frame image is defined as the motion detection area Ea1. It shows how it was set. FIG. 10 is a diagram showing another example of the motion detection area set in the immediately preceding frame image, and shows a state in which a face area having a size larger than that in FIG. 9 is set as the motion detection area Ea2. . Although FIG. 9 and FIG. 10 show one motion detection area, in actuality, the face area selection unit 18 selects a face area having a predetermined number of motion detection areas, The motion detection unit 15 sets each face area as a motion detection area.

  Then, the motion detection unit 15 performs pattern matching with the current frame image for the motion detection areas Ea1 and Ea2 set in the immediately preceding frame image as shown in FIGS. Here, the pattern matching is performed by setting a plurality of macro blocks in the motion detection area and obtaining the motion vector for each macro block in order to improve accuracy. For example, in the example shown in FIG. 9, four macroblocks B are set in the motion detection area Ea1. On the other hand, in the example shown in FIG. 10, 25 macroblocks B are set in the motion detection area Ea2, and the number of macroblocks B corresponding to the size of the motion detection area is set. Note that the number of macroblocks set in the motion detection area may be the same, and the size of the macroblock can be set as appropriate.

  As shown in FIG. 8, the face detection unit 17 detects the face by processing the image data digitized by the AFE 13 and recorded in the RAM 27, and temporarily records the face detection result in the RAM 27. For example, the face detection unit 17 detects the position of the face in the live view image by the template matching method, and detects the position of the feature point of the face such as the eyes, nose, mouth, etc. Etc. are detected. By the face detection by the face detection unit 17, for example, presence / absence of a face in the live view image, face position, face size, face orientation, face angle (tilt), and the like are obtained as face detection results. As the face detection result, at least the latest face detection result is held for each detected face.

  The face area selection unit 18 selects face areas corresponding to the number of motion detection areas, and switches and sets a motion detection area for performing motion detection with the next frame based on the selection result. In the present embodiment, the face area selection unit 18 selects a face area of a face with high priority based on the priority scored by the priority scoring unit 293 of the controller 29 for each face detected by the face detection unit 17. The face area of each selected face is set as a motion detection area.

  The image processing unit 16 reads the image data once recorded in the RAM 27, performs various image processes on the image data, and performs a process of converting the image data into image data suitable for recording or display. For example, when recording image data of a photographed image or displaying recorded image data, image data compression processing or decompression processing based on a JPEG (Joint Photographic Experts Group) method or the like is performed. The image data processed by the image processing unit 16 is output to the recording medium I / F 19 and recorded on the recording medium 21 or is output to the video encoder 22 and displayed on the display unit 24.

  The video encoder 22 sends the image data converted for display to the display driver 23. For example, in the shooting mode, an image captured from the image sensor 12 for each frame and subjected to image processing by the image processing unit 16 is switched and displayed on the display unit 24 in units of frames to display a live view image. On the other hand, in the reproduction mode, the captured image read from the recording medium 21 and image-processed by the image processing unit 16 is displayed on the display unit 24. The video encoder 22 outputs image data for display to an external device connected to the video signal output terminal 25 as necessary. The display unit 24 displays various setting information of the digital camera 1 in addition to the captured image and the live view image, and is realized by a display device such as an LCD (Liquid Crystal Display) or an EL display (Electroluminescence Display). Is done.

  The recording medium I / F 19 performs writing of image data converted for recording, reading of recorded image data, and the like with respect to the recording medium 21 detachably held by the recording medium holding unit 20. . The recording medium 21 is a memory card such as an xD-picture card (registered trademark) or a compact flash (registered trademark) card.

  The operation unit 26 is for accepting various operations by the user, such as an instruction of shooting timing, a setting operation of a shooting mode or a playback mode, a setting operation of shooting conditions, and the like, and notifies an operation signal to the controller 29. It is realized with button switches, dials, various sensors, etc., to which various functions are assigned. The operation unit 26 includes the release switch 3, the power switch 4, the menu switch 5, the cross switch 6, the OK switch 7 and the mode dial 8 of FIG.

  The ROM 28 previously records various camera programs for operating the digital camera 1 and realizing various functions of the digital camera 1, data used during the execution of the camera program, and the like. The RAM 27 is used as a working memory for the image processing unit 16 and the controller 29. For example, image data or the like from the AFE 13 is temporarily recorded and used for generating image data of a live view image to be displayed on the display unit 24 or for recording a captured image on the recording medium 21. Used.

  The controller 29 reads out and executes a camera program from the ROM 28 in response to an operation signal from the operation unit 26, and performs operation control and memory control of each unit constituting the digital camera 1 to control the entire operation of the digital camera 1. To control. In addition, processing such as AF (automatic focus), AE (automatic exposure), and AWB (automatic white balance) is performed. The controller 29 includes a shooting start instruction unit 291, a priority scoring unit 293 as a difficulty level evaluation unit, a motion detection candidate setting unit 295 as a face region setting unit, and a face area determination unit 297 as a face region determination unit. including. The shooting start instruction unit 291 instructs the start timing of the shooting process. The priority scoring unit 293 scores the priority of the detected face based on the face detection result. In this embodiment, the priority scoring unit 293 uses the face size, face position, face orientation, face orientation change, face overlap, face angle, and face angle change as evaluation parameters. Find the scoring value of the evaluation parameter. Then, the priority scoring unit 293 weights and adds the scoring values of the respective evaluation parameters obtained using the weighting factors set in advance for the respective evaluation parameters to obtain the priority. In addition, as an evaluation parameter, it is not limited to what was illustrated. For example, the face moving speed or the face moving speed may be determined based on the face detection result, and these values may be used as evaluation parameters. The motion detection candidate setting unit 295 sets the next motion detection candidate based on the face area based on the face detection result and the face area based on the motion detection result. The face area determination unit 297 determines the face area in the live view image of the current frame.

  Next, a processing procedure performed by the digital camera 1 will be described. FIG. 11 is a flowchart illustrating a procedure of basic processing performed by the digital camera 1. When the power is turned on, the digital camera 1 performs processing according to the mode selected by the mode dial 8. That is, as shown in FIG. 11, when the currently selected mode is the shooting mode (step a1: Yes), the controller 29 images the subject image formed on the image sensor 12 (live view image). ) (Step a3), and the process proceeds to face area detection processing (step a5). In the face area detection process, the face area in the live view image of the current frame captured in step a3 is detected by face detection and motion detection, and the face area in the live view image is determined.

  When the face area detection process is completed, the controller 29 subsequently performs a process of displaying the live view image on the display unit 24 (step a7). At this time, a live view image in which a face frame indicating a face is displayed in the face area determined in step a5 is displayed on the display unit 24. If there is a face area determined in step a5 (step a9: Yes), the controller 29 sets the imaging condition based on the determined face area and performs processing such as AF, AE, and AWB ( Step a11). Specifically, the controller 29 sets the imaging condition using the latest face detection result obtained for the face in the face area determined in step a5. That is, among the determined face areas, the face area based on the face detection result (the face area of the face that has been successfully detected this time) is used, and the face area based on the motion detection result (for the current face detection). For the face area of the failed face), the imaging condition is set using the face detection result when the face succeeded last time. On the other hand, if there is no face area determined in step a5 (step a9: No), the controller 29 performs AF, AE, AWB, etc. in the normal range (for example, the entire screen) (step a13).

  Then, until the release switch 3 is pressed down one step and the first release switch is turned on (step a15: No), the process returns to step a3 and the process is repeated for each frame.

  When the release switch 3 is pressed down one step and the first release switch is turned on (step a15: Yes), the controller 29 then continues to detect the subject imaged on the image sensor 12 in the same manner as in step a3. An image is captured (step a17). Thereafter, the process proceeds to face detection processing (step a19). In the face area detection process, the face area in the live view image of the current frame captured in step a17 is detected by face detection and motion detection, and the face area in the live view image is determined.

  Then, after the face detection process, the controller 29 performs a process of displaying the live view image on the display unit 24 (step a21). At this time, a live view image in which a face frame indicating a face is displayed in the face area determined in step a19 is displayed on the display unit 24. If there is a face area determined in step a19 (step a23: Yes), the controller 29 sets the imaging condition based on the determined face area and performs processing such as AF, AE, and AWB ( Step a25). Specifically, the controller 29 sets the imaging condition using the latest face detection result obtained for the face in the face area determined in step a19, as in step a11. On the other hand, if there is no face area determined in step a19 (step a23: No), the controller 29 performs processing such as AF, AE, AWB, etc. in the normal range (step a27).

  Then, until the release switch 3 is pressed in two stages and the second release switch is turned on (step a29: No), the process returns to step a17 and the process is repeated for each frame.

  If the release switch 3 is depressed in two stages and the second release switch is turned on (step a29: Yes), the process proceeds to the photographing process (step a31). Specifically, the imaging start instruction unit 291 instructs the start of the imaging process with the timing when the second release switch is turned on as the imaging timing. Thereby, a photographing process is started, and image data of the photographed image is generated. The generated image data of the captured image is recorded on the recording medium 21. Note that when this shooting process is started, the display of the live view image is temporarily stopped. The display of the live view image is restored after the image data transfer processing and image processing are completed after exposure.

  Subsequently, the process proceeds to step a33, and the controller 29 determines the end of the shooting mode. When the shooting mode is to be ended (step a33: Yes), the process proceeds to step a43. If the shooting mode is not terminated (step a33: No), the process returns to step a3.

  On the other hand, when the currently selected mode is not the shooting mode (step a1: No) and in the playback mode (step a35: Yes), the controller 29 has been shot in the past and recorded on the recording medium 21. A process for displaying a list of image data of still images and moving images, for example, in a thumbnail format is performed, and a reproduced image is selected from the list in accordance with a user operation (step a37). Then, the controller 29 performs a process of displaying the selected reproduction image on the display unit 24 (step a39).

  Subsequently, the process proceeds to step a41, and the controller 29 determines the end of the reproduction mode. When the reproduction mode is to be ended (step a41: Yes), the process proceeds to step a43. If the reproduction mode is not terminated (step a41: No), the process returns to step a37.

  In step a43, the controller 29 determines whether to end the basic process. For example, when the power switch 4 is pressed and an instruction to turn off the power is given, this processing is finished (step a43: Yes). On the other hand, if not finished (step a43: No), the process returns to step a1.

  Next, the face area detection process performed in step a5 and step a19 in FIG. 11 will be described. FIG. 12 is a flowchart showing a detailed processing procedure of the face area detection processing. In the face area detection process, first, the controller 29 determines whether there is a motion detection candidate. Here, a motion detection candidate is set in step b15 in FIG. For this reason, there is no motion detection candidate in the face area detection process performed first after the photographing mode is selected (step b1: No), and the process proceeds to step b9. That is, the face detection unit 17 detects a face in the current frame image (step b9), and records the face detection result in the RAM 27 (step b11).

  Subsequently, the priority scoring unit 293 executes a priority scoring process (step b13). FIG. 13 is a flowchart showing a detailed processing procedure of the priority scoring process. This priority scoring process is performed for each of the faces detected in step b9 in FIG. 12, and the priority scoring unit 293 scores the priority of each face based on the face detection result for that face. .

  That is, the priority scoring unit 293 first scores the face size based on the face detection result of step b9 in FIG. 12 (step c1). Larger faces are more important than smaller faces. Therefore, for example, the priority scoring unit 293 sets the scoring value larger as the face is larger.

  Subsequently, the priority scoring unit 293 scores the face position (step c3). The closer the face position is to the center of the field angle range, the higher the importance. On the other hand, when the face position is located at the end of the angle of view range, the face is likely to be framed out in the next frame, so the importance is low. Therefore, for example, the priority scoring unit 293 sets the scoring value larger as the face position is closer to the center of the angle of view range.

  Subsequently, the priority scoring unit 293 scores the face orientation (step c5). When the face is facing away from the front direction, there is a high possibility that the face is facing away from the front direction in the next frame, and the probability of face detection failure is high. Therefore, for example, the priority scoring unit 293 decreases the scoring value when the face is facing the front, and sets the scoring value larger as the face is deviated from the front.

  Subsequently, the priority scoring unit 293 scores the face orientation change (step c7). Here, in the RAM 27, the results for the last several frames are stored. The priority scoring unit 293 calculates a change in the face direction with reference to the face detection results for the past several frames held in the RAM 27, and scores the calculated change in direction. That is, if the face orientation has changed in the past several frames, the possibility that the face orientation will change in the next frame is high. And as the orientation change is larger, the possibility that the face orientation will change greatly in the next frame is higher, so the possibility of face detection failure is also higher. Therefore, for example, the priority scoring unit 293 sets the scoring value larger as the face direction change is larger.

  Subsequently, the priority scoring unit 293 estimates a case where faces overlap in the next frame (step c9). FIG. 14 is a diagram for explaining face overlap estimation, and schematically shows an example of three live view images in which the faces F11 to F13 of three persons are shown in time series. Here, paying attention to the faces F11 and F12, the faces F11 and F12 approach between the frame I31 in FIG. 14A and the frame I33 in FIG. 14B, and each frame I35 in FIG. Faces F11 and F12 overlap. In the overlap estimation, such a state is estimated. That is, the priority scoring unit 293 refers to face detection results for the past several frames. The priority scoring unit 293 determines the moving direction (direction in which the direction changes) and the moving speed (moving amount) based on the position, orientation, and size of each face, and estimates the overlap of the faces. For example, in the frame I31 in FIG. 14A, the positions of the faces F11 to F13 are separated, and it is estimated that the faces F11 to F13 do not overlap in the next frame. On the other hand, in the frame I33 in FIG. 14B, it is estimated that the faces F11 and F12 are approaching and that the faces F11 and F12 overlap in the next frame I35 based on the moving direction and moving speed.

  Then, as shown in FIG. 13, the priority scoring unit 293 scores the face overlap based on the overlap estimation result (step c11). Specifically, the priority scoring unit 293 sets a small scoring value for the face that is hidden behind when it overlaps based on the face detection result for each face estimated to overlap. As estimated, if the face overlaps with another face in the next frame and hides behind the other face, there is no need to display the face frame, and there is no need to control exposure or focus for that face. This is because the importance is low. Here, when the faces overlap, which can be hidden behind can be determined by the size of the face. That is, a large size face exists on the near side, and when the faces overlap, it is considered that the small size face is hidden behind the large size face. For example, in the frame I35 in FIG. 14C, the small face F12 is hidden behind the large face F11. The score value of such a face is set smaller than the score values of other faces.

  Subsequently, the priority scoring unit 293 scores the face angle (step c13). When the face is tilted with respect to the vertical direction of the live view image, there is a high possibility that the face is tilted even in the next frame, and the probability of face detection failure is high. Therefore, for example, the priority scoring unit 293 decreases the scoring value when the face is not inclined, and sets the scoring value larger as the face has a larger angle.

  Subsequently, the priority scoring unit 293 scores the face angle change (step c15). The priority scoring unit 293 calculates the change in the face angle with reference to the face detection results for the past several frames, and scores the calculated angle change. That is, if the face angle has changed in the past few frames, the face angle is likely to change in the next frame. And as the angle change is larger, there is a higher possibility that the face angle will change greatly in the next frame. Therefore, for example, the priority scoring unit 293 sets the scoring value larger as the face angle change is larger.

  Subsequently, the priority scoring unit 293 performs weighting for each scoring value (step c17). For the weighting performed here, for example, a weighting factor is set in advance for each evaluation parameter of face size, face position, face orientation, face orientation change, face overlap, face angle, and face angle change. Every scoring value is multiplied by a weighting factor. The weighting factor can be set as appropriate according to the importance of the evaluation parameter. For example, if the weighting coefficient of the face direction and the face angle, which are evaluation parameters that cause the detection accuracy of the face detection unit 17 to decrease, is set large, the priority of the face having a high score value is increased. be able to.

  And the priority scoring part 293 calculates the sum total of each weighted scoring value as a priority (step c19). Thereafter, the process returns to step b13 in FIG. 12 and proceeds to step b15. The priority scoring process described above is an example, and the scoring method and evaluation parameters are not limited to this, and can be set as appropriate according to face detection specifications and the like.

  In the subsequent step b15 of FIG. 12, the motion detection candidate setting unit 295 executes a motion detection candidate setting process. FIG. 15 is a flowchart illustrating a detailed processing procedure of the motion detection candidate setting process.

  In the motion detection candidate setting process, the motion detection candidate setting unit 295 first sets a face area based on the motion detection result (step d1). As described above, in the face area detection process performed first after the shooting mode is selected, the motion detection in step b15 in FIG. 12 has not been performed yet. For this reason, the face area is not set even in step d1. On the other hand, as described later, in the second and subsequent face area detection processing, when it is determined that there is a motion detection candidate in step b1 and motion detection is performed in step b5, the motion detection candidate setting unit 295 performs this step. At d1, the position of each motion detection area in the current frame is calculated based on the motion vector of each motion detection area calculated as a result of motion detection, and set as a face area based on the motion detection result. Subsequently, the motion detection candidate setting unit 295 sets a face area detected as a result of face detection as a face area based on the face detection result (step d3).

  Then, the motion detection candidate setting unit 295 sets the face area based on the motion detection result in the current frame and the face area based on the face detection result as the next motion detection candidate (step d5).

  Subsequently, the motion detection candidate setting unit 295 compares the position of the face area based on the motion detection as the next motion detection candidate with the position of the face area based on the face detection result. Then, the motion detection candidate setting unit 295, when any face area position based on the face detection result matches the position of any face area based on the motion detection result, that is, the face area based on the motion detection result. And the face area based on the face detection result overlap (step d7: Yes), the face area based on the motion detection result is excluded from the next motion detection candidates (step d9). ). Thereafter, the process returns to step b15 in FIG. 12 and proceeds to step b17.

  In step b17, the next motion detection candidate set as a result of the motion detection candidate setting process in step b15 is determined as the face area in the current frame. Thereafter, the process returns to step a5 in FIG. 11 and proceeds to step a7. Or it returns to step a19 and moves to step a21. As a result, a face frame is displayed in the face area detected by the face area detection process on the live view image displayed in step a7 or step a21. The detected face area is used for exposure and focus control.

  Further, in the face area detection process performed after the shooting mode is selected for the second and subsequent times, as shown in FIG. 12, if the next motion detection candidate is set in step b15 in the previous face area detection process ( Step b1: Yes), the process moves to step b3, and the face area selection unit 18 executes the face area selection process. FIG. 16 is a flowchart showing a detailed processing procedure of the face area selection processing.

  In the face area selection process, the face area selection unit 18 first compares the number of set motion detection candidates with a preset number N of motion detection areas. If the number of motion detection candidates is equal to or less than the number of motion detection areas N (step e1: No), the face area selection unit 18 selects all motion detection candidates (step e3). And it returns to step b3 of FIG. 12, and moves to step b5 after that.

  If the number of motion detection candidates is greater than the number N of motion detection areas (step e1: Yes), the face area selection unit 18 proceeds to step e5. If the number L of face areas based on the motion detection result is “0” (step e5: Yes), the face area selection unit 18 proceeds to step e7. That is, if the number L of face areas based on the motion detection result is “0”, the set motion detection candidates are all face areas based on the face detection result. In step e7, the face area selection unit 18 selects N face areas from the face areas based on the face detection result in descending order of the priority scored for the face. And it returns to step b3 of FIG. 12, and moves to step b5 after that.

  If the number L of face areas based on the motion detection result is not “0” (step e5: No), the face area selection unit 18 proceeds to step e9. If the number of face areas based on the face detection result is “0” (step e9: Yes), the face area selection unit 18 proceeds to step e11. That is, when the number of face areas based on the face detection result is “0”, the set motion detection candidates are all face areas based on the motion detection result, and the number thereof is N. In step e11, the face area selection unit 18 selects all face areas based on the motion detection result. And it returns to step b3 of FIG. 12, and moves to step b5 after that.

  If the number of face areas based on the face detection result is not “0” (step e9: No), the face area selection unit 18 proceeds to step e13. Then, the face area selection unit 18 selects all the face areas based on the motion detection results, and if less than N, selects the face areas based on the face detection results in descending order of priority. Select NL pieces. And it returns to step b3 of FIG. 12, and moves to step b5 after that.

  In step b5, the motion detection unit 15 performs motion detection in the current frame image using each face area selected in the face area selection processing in step b3 as a motion detection area, and then records the motion detection result in the RAM 27. (Step b7). Then, it moves to step b9.

  As described above, according to the present embodiment, a face in a live view image can be detected by face detection, and the priority of each face can be scored according to the face detection result. For example, when the movement of a person is intense, the priority can be set high for a face that is assumed to have a high detection difficulty level in the next frame. As a result of face detection, if more faces than the number of motion detection possible (number of motion detection areas) are detected in the live view image, a face area including a face with high priority is selected and moved. Detection can be performed. That is, it is possible to preferentially select a face that is assumed to have a high degree of detection difficulty for face detection, and to detect the movement of the face area by motion detection, so that a continuous image is not increased without increasing the processing load. It is possible to track stably without losing sight of the face that appears inside.

  In the present embodiment, a face frame indicating a face can be displayed in the finally determined face area on the live view image. Therefore, stable and easy-to-view face frame display can be realized, and problems such as flickering by repeatedly displaying and hiding the face frame can be prevented.

  In the embodiment described above, a face area having a preset number of motion detection areas is selected as the motion detection area, and motion detection is performed for the selected motion detection area. When selecting a face area based on the face detection result as a motion detection area, select the face area as the motion detection area in order from the highest priority scored for each face based on the face detection result. It was decided to. On the other hand, a configuration may be adopted in which a face area of a face with a low priority is selected from the faces detected by face detection, and a setting is made so that motion detection is not performed for the selected face area.

  In the above-described embodiment, a face area that can be detected by motion detection but fails to detect a face is always selected as a motion detection area. On the other hand, the motion detection area may be set in consideration of the reliability of the motion detection result. That is, for example, when the number of matching failures is larger than the number of macroblocks B (see FIGS. 9 and 10) set in the motion detection area, or the direction of the motion vector for each obtained macroblock B is uniform. If there is no, it is considered that the reliability of the motion detection result is low. In such a case, the corresponding face area may be removed from the next motion detection area. If it demonstrates with reference to FIG. 7, suppose that the reliability of the motion detection result performed, for example using the face area of the face F2 as a motion detection area was low. In this case, the face area of the face F2 is removed from the target of motion detection performed with the next frame. And it is good also as a motion detection area by selecting a face with high priority among the face areas of the face F3 and the face F4 which are face areas based on a face detection result. In the illustrated example, for example, the face F4 is selected as the motion detection area.

  In the above-described embodiment, the face detection and the motion detection are performed for each frame, but may be performed at a predetermined frame interval. Also, the frame intervals for performing face detection and motion detection can be set individually as appropriate. For example, motion detection can be performed every frame, and face detection can be performed every few frames.

  In addition, the face area selection unit 18, the image processing unit 16, the imaging start instruction unit 291, the priority scoring unit 293, and the motion detection candidate setting unit 295 that constitute the controller 29 may be realized by hardware, or may be predetermined. It is good also as implement | achieving as software by running this program. When realized as software, for example, a program for realizing part or all of the processing shown in FIGS. 12, 13, 15, 16, etc. is recorded in the ROM. The controller 29 reads out and executes this program, thereby realizing the configuration of the face area selection unit 18, the image processing unit 16, the imaging start instruction unit 291, the priority scoring unit 293, and the motion detection candidate setting unit 295. It may be.

  In the above-described embodiment, the example in which the tracking device of the present invention is applied to a digital camera has been described. However, the application target is not limited to a digital camera, and is applied to a camera attached to a mobile phone or a camera attached to a PC. May be. Further, the present invention can also be applied to the case where a moving image showing a person's face is reproduced using a personal computer or the like.

It is a rear view of a digital camera. It is the figure which showed an example of the live view image updated and displayed for every frame along a time series. It is a figure which shows an example of the face detection result performed about the live view image in which five faces are reflected. It is a figure which shows an example of the face detection result and motion detection result which were performed about the live view image in which five faces are reflected. It is a figure explaining the setting principle of a motion detection area. It is another figure explaining the setting principle of a motion detection area. It is another figure explaining the setting principle of a motion detection area. It is a schematic block diagram which shows the structural example of a digital camera. It is a figure which shows an example of the motion detection area set to the immediately preceding frame image. It is a figure which shows the other example of the motion detection area set to the immediately preceding frame image. It is a flowchart which shows the procedure of the basic process which a digital camera performs. It is a flowchart which shows the detailed process sequence of a face area detection process. It is a flowchart which shows the detailed process sequence of a priority scoring process. It is a figure explaining the overlap estimation of a face. It is a flowchart which shows the detailed process sequence of a motion detection candidate setting process. It is a flowchart which shows the detailed process sequence of a face area selection process.

DESCRIPTION OF SYMBOLS 1 Digital camera 2 Camera body 11 Imaging optical system 12 Imaging element 13 AFE
14 frame memory 15 motion detection unit 16 image processing unit 17 face detection unit 18 face area selection unit 19 recording medium I / F
DESCRIPTION OF SYMBOLS 20 Recording medium holding | maintenance part 21 Recording medium 22 Video encoder 23 Display driver 24 Display part 25 Video signal output terminal 26 Operation part 3 Release switch 4 Power switch 5 Menu switch 6 Four-way controller 7 OK switch 8 Mode dial 27 RAM
28 ROM
29 controller 291 imaging start instruction unit 293 priority scoring unit 295 motion detection candidate setting unit 297 face area determination unit

Claims (9)

A tracking device that tracks faces appearing in successive images,
A face detector that sequentially processes the successive images to detect a plurality of faces in the images;
A face area setting unit that sets a face area including each face detected by the face detection unit for each face;
Based on the face detection result by the face detection unit, a difficulty level evaluation unit that evaluates the detection difficulty level of each face detected by the face detection unit,
Based on the evaluation result by the difficulty level evaluation unit, a predetermined number of the face regions are selected from the face regions including the faces set by the face region setting unit in descending order of difficulty of detection of the faces. A face area selection unit to be
A motion detection target setting unit that sets a target region for motion detection with respect to the face region selected by the face region selection unit;
A motion detection unit that detects a motion of the target region set by the motion detection target setting unit between adjacent images;
A tracking device comprising: The face detection unit outputs at least one of a face size, a face position, a face orientation, and a face tilt as the face detection result;
The difficulty level evaluation unit, based on the face detection result of each face detected by the face detection unit, at least the size of the face, the position of the face, the direction of the face, the inclination of the face, the change of the face direction, The tracking device according to claim 1, wherein the degree of difficulty in detecting each face is evaluated using one or more of a change in face inclination, a face moving speed, and a face moving direction as evaluation parameters.   3. The difficulty evaluation unit estimates the possibility that each detected face overlaps with another face, and evaluates the detection difficulty of each face using the estimation result as the evaluation parameter. The tracking device described in 1.   The difficulty level evaluation unit determines a moving speed and / or moving direction of the face from the face detection result of each face, and estimates the possibility that each face overlaps with another face based on the determination result. The tracking device according to claim 3. The difficulty level evaluation unit weights the evaluation parameter using a weighting factor set in advance for each evaluation parameter, and evaluates the detection difficulty level of each face based on the weighted evaluation parameter. The tracking device according to claim 2 .   A face region determining unit that determines a face region in the image based on a face region including each face detected by the face detecting unit and a motion of the target region detected by the motion detecting unit; The tracking device according to claim 1. A display processing unit for switching the continuous images and performing display processing on the display unit;
The tracking apparatus according to claim 6, wherein the display processing unit displays a face frame indicating a face in the image according to the face region in the image determined by the face region determination unit. An imaging unit that sequentially images the subject for each frame and sequentially generates the continuous images;
A shooting instruction section for giving shooting instructions;
An imaging condition setting unit that sets an imaging condition of the imaging unit using the latest face detection result detected by the face detection unit for the face of the face region in the image determined by the face region determination unit;
The tracking device according to claim 6, further comprising: A tracking method for tracking a face that appears in successive images,
A face detection step of sequentially processing the continuous images to detect a plurality of faces in the images;
A face area setting step for setting a face area including each face detected in the face detection step for each face;
Based on the face detection result in the face detection step, a difficulty level evaluation step for evaluating the detection difficulty level of each face detected in the face detection step;
Based on the evaluation result of the difficulty level evaluation step, a predetermined number of the face regions are selected from the face regions including the faces set in the face region setting step in descending order of difficulty of detection of the faces. A face area selection step to be performed;
A motion detection target setting step for setting as a target region for motion detection for the face region selected by the face region selection step;
A motion detection step of detecting a motion of the target region set in the motion detection target setting step between adjacent images;
The tracking method characterized by including.

Images