JP5451364B2 - Subject tracking device and control method thereof - Google Patents

Description

  The present invention relates to a subject tracking device and a control method thereof.

  An image processing technique that detects a specific subject in one frame of an image that is sequentially supplied in time series and tracks the detected subject in subsequent frames is very useful. For example, techniques for detecting and tracking a human body region (eg, a face region) in a moving image include many fields such as a conference call, a man-machine interface, security, a monitor system for tracking a specific subject, and image compression. Can be used in Digital cameras and digital video cameras equipped with this image processing technology are known to perform exposure control and focus control so that the subject specified in the image displayed on the electronic viewfinder is properly captured. Yes.

  For example, Patent Document 1 discloses a photographing apparatus that detects a face from a photographed image, focuses on the face, and photographs the face with an optimal exposure. Japanese Patent Application Laid-Open No. H10-228688 discloses performing tracking processing for detecting a face detected in a certain frame in a subsequent frame.

  As a method for tracking a specific subject in a subsequent frame, a method using a template matching method as disclosed in Patent Document 2 is known. In template matching, an image of a subject to be tracked (template image or reference image) is extracted from a frame with a moving image (referred to as a reference frame), and a region having the highest degree of correlation with the template image is searched in subsequent frames. Technology. The subject corresponding to the template image can be tracked by estimating the searched area as the position of the template image in the subsequent frame.

JP 2005-318554 A JP 2001-60269 A

  In the subject tracking method using template matching, the subject is tracked based on the degree of correlation between the frame image for tracking the subject and the reference image (template image). Here, when a region similar to the reference image but different from the subject to be tracked exists in the frame image, the region may be erroneously detected as the subject. This problem is particularly likely to occur when the appearance of the subject in the frame image changes from the reference image. Therefore, the assumption that the position of the subject does not change greatly between two frame images that are continuous in time series is introduced, and among the regions having a high degree of correlation, the region where the moving distance between the images is large is the subject region. It can be considered that it is not. As a result, when the movement of the subject is small, the possibility of erroneously detecting the similar region as the subject region will be reduced.

  However, when the movement of the subject is fast, the change in the position of the subject area between two frame images that are continuous in time series is large. Therefore, when the movement of the subject is fast, the possibility of erroneous detection is rather increased by introducing the above assumption. May increase.

  The present invention has been made in view of the problems of the prior art. The present invention relates to a subject tracking device that performs subject tracking based on the degree of correlation between a reference image and an input image, and a control method thereof, in which the accuracy of subject tracking is improved when an object similar to the subject exists or when the subject moves quickly. The purpose is to improve.

  The above-described object is a subject tracking device that tracks a specific subject in an image over a plurality of input images input in time series, and is set to a reference image and an input image registered in a registration unit. A degree-of-correlation calculating means for obtaining a degree of correlation with each of a plurality of reference regions, an extracting means for extracting a feature amount of a standard image, a degree-of-match determination means for determining a degree of matching of an input image with a feature amount, A distance calculating unit that calculates a distance between each of the reference regions and a predetermined reference position in the input image, a correlation degree, a distance, and a degree determined by the matching degree determining unit for each of the plurality of reference regions And an evaluation value calculating means for calculating an evaluation value using the reference value, and a determining means for determining that the reference area having the largest evaluation value among the plurality of reference areas is the area of the specific subject. Is determined by the determination means most recently The evaluation value calculation means calculates the evaluation value as a function value of the correlation degree and the distance, and the degree of coincidence determination means determines the degree of contribution of the distance to the evaluation value. The object tracking device is characterized in that the evaluation value is dynamically calculated so that the evaluation value is dynamically changed so as to increase as the degree of determination increases and decrease as the degree of determination by the coincidence determination unit decreases.

  According to the present invention, in a subject tracking device that performs subject tracking based on the degree of correlation between a reference image and an input image and a control method thereof, subject tracking can be performed when an object similar to the subject exists or when the subject moves quickly. Accuracy can be improved.

1 is a block diagram illustrating an example of a functional configuration of an imaging apparatus as an example of a subject tracking apparatus according to an embodiment of the present invention. 6 is a flowchart showing subject tracking processing according to the embodiment of the present invention. Explanatory drawing of the subject tracking which concerns on embodiment of this invention.

  Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. The description of the embodiment described below is intended to help understanding of the present invention, and does not limit the scope of the present invention, and various modifications are possible within the scope of the present invention. is there. In the following embodiments, a configuration in which the present invention is applied to an imaging apparatus will be described. However, in the present invention, an imaging function is not essential, and any source of images to be subject-tracked can be supplied. Good. Therefore, the present invention can be widely applied to an image processing apparatus that reads a recorded image, reproduces and displays it, and receives and processes an image from the outside. Such an image processing apparatus includes various devices such as a personal computer, a portable information terminal, a mobile phone, and a media player.

(First embodiment)
FIG. 1 is a block diagram illustrating a functional configuration example of an imaging apparatus 100 as an example of subject tracking according to the first embodiment of the present invention. The lens unit 101 is an imaging optical system, and generally includes a plurality of lenses including a focus adjustment lens. The lens unit 101 forms an optical image of the subject on the imaging surface of the image sensor 102. The imaging element 102 is a photoelectric conversion element such as a CCD image sensor or a CMOS image sensor, and converts the subject optical image into an electrical signal in units of pixels. This electrical signal is an analog video signal indicating a subject image captured by the image sensor 102. The image sensor 102 can output video signals in time series at a predetermined frame rate by continuously repeating imaging and output. In the following description, “image” is synonymous with a one-frame video signal.

  The image output from the image sensor 102 is subjected to analog signal processing such as correlated double sampling (CDS) in the analog signal processing unit 103, and then converted into a digital data format in the A / D conversion unit 104. The data is input to the unit 105 and the image processing unit 106.

  The control unit 105 is a CPU (Central Processing Unit), a microcontroller, or the like, and controls the operation of the entire imaging apparatus 100. Specifically, the control unit 105 controls each unit of the imaging apparatus 100 by developing program codes stored in a ROM (Read Only Memory) in a work area of a RAM (Random Access Memory) and sequentially executing the program codes. . Note that the control unit 105 may include a single CPU or a plurality of CPUs, and the functions realized by the control unit 105 may be distributed and processed by a plurality of CPUs.

  The image processing unit 106 applies image processing such as gamma correction and white balance processing (so-called development processing) to the input digital format image. In addition to the image processing, the image processing unit 106 also executes image processing using information on a specific subject area in the image supplied from the subject tracking unit 110 described later.

  The image output from the image processing unit 106 is displayed on the display unit 107 which is an LCD or an organic EL display, for example. The imaging apparatus 100 can cause the display unit 107 to function as an electronic viewfinder (EVF) by displaying the moving image captured by the imaging element 102 on the display unit 107 in real time. Further, when the display unit 107 functions as an electronic viewfinder, the display unit 107 can display information indicating the position of the subject area tracked by the subject tracking unit 110 described later together with the captured image.

  Further, the image output from the image processing unit 106 is recorded by the recording unit 108 on, for example, a removable memory card. Note that the image recording destination may be the built-in memory of the imaging apparatus 100 or an external apparatus (none of which is shown) that is communicably connected via a communication interface. Note that the image processing unit 106 can optimize and output the image output to the display unit 107 and the image output to the recording unit 108 for display and recording, respectively. The image processing unit 106 can optimize, for example, the resolution (number of pixels), the signal format (RGB format, YCrCb format), and the data format of the image according to the application. The image processing unit 106 also executes image encoding processing and decoding processing as necessary.

  The subject designating unit 109 is an input interface such as a touch panel provided on the display unit 107 and keys and buttons provided on the housing of the imaging apparatus 100. For example, the user can designate a subject to be tracked by designating a desired subject region from the input image displayed on the display unit 107. There is no particular limitation on a method for designating an arbitrary area from an image using a touch panel, keys, buttons, or the like, and a well-known method can be adopted. Further, for example, when the imaging apparatus 100 has a subject detection function and an index such as a frame indicating the detected subject area is superimposed on the captured image, the subject can also be selected by selecting a desired subject frame. Can be specified.

  The subject tracking unit 110 performs image matching by template matching between the image output from the image processing unit 106 and a reference image based on the subject image in the input image specified by the subject specifying unit 109 or an updated reference image. The object image position inside is detected. The subject tracking unit 110 implements the subject tracking function by applying this detection process to a plurality of images input in time series from the image processing unit 106 and having different imaging times.

  In the subject tracking unit 110, the feature amount extraction unit 116 and the feature matching degree determination unit 117 calculate the reliability of the reference image registered in the reference image registration unit 111. The evaluation value calculation unit 115 dynamically changes the degree that the distance calculated by the distance calculation unit 114 contributes to the evaluation value according to the calculated reliability.

  The reference image registration unit 111 registers partial images of a plurality of images having different times from the image supplied by the image processing unit 106 as a reference image. The registered reference image is used by the correlation degree calculation unit 113. The reference image registered by the reference image registration unit 111 is registered based on the region specified by the subject specifying unit 109 at the start of subject tracking, and then sequentially by the subject region extracted by the subject extracting unit 118 during subject tracking. Updated.

  In the reference area setting unit 112, a part of the input image sequentially supplied in time series by the image processing unit 106 is set as a reference area used in the correlation calculation unit 113 and the distance calculation unit 114. The size of the reference area corresponds to the size of the standard image registered by the standard image registration unit 111. The reference area is sequentially set with respect to one input image supplied by the image processing unit 106 while shifting the position in units of pixels in the horizontal and vertical directions. The area (search area) in which the reference area is set in the input image is a partial area such as a predetermined range centered on the position of the area determined as the subject area in the immediately preceding input image, even in the entire image. May be.

  The correlation degree calculation unit 113 calculates the degree of correlation between the standard image registered by the standard image registration unit 111 and the image of the reference area set by the reference area setting unit 112. As described above, since the reference region setting unit 112 sequentially sets the reference region while shifting the position in the same input image (search region), the reference image has the highest correlation with the reference image in the input image (search region). A high region can be estimated. The degree of correlation can be obtained, for example, as the sum of differences between pixel values at corresponding positions in the reference region image and the base image, and the smaller the difference sum value, the higher the degree of correlation. There are various methods for calculating the degree of correlation between two images, and any method can be applied to this embodiment.

  The distance calculation unit 114 calculates the distance between the position of the reference area set by the reference area setting unit 112 and a predetermined reference position in the input image. The predetermined reference position is the position of the subject area extracted by the subject extraction unit 118 in the input image immediately before subject tracking. Alternatively, the predetermined position may be a position of the subject area in the current input image predicted based on past subject area motion information obtained by subject tracking. Here, the “position of the subject area” can be arbitrarily set, and may be, for example, the position of the center of gravity of the subject area or the position of one vertex of the subject area.

  The evaluation value calculation unit 115 calculates an evaluation value for each reference area according to the correlation degree calculated by the correlation degree calculation unit 113 and the distance calculated by the distance calculation unit 114. When calculating the evaluation value, the evaluation value calculation unit 115 changes the degree that the distance calculated by the distance calculation unit 114 contributes to the evaluation value according to the degree of matching determined by the feature matching level determination unit 117 described later. Let

  The feature amount extraction unit 116 extracts a feature amount from the reference image registered in the reference image registration unit 111. For example, the feature amount extraction unit 116 generates a histogram for predetermined component values of pixels included in a predetermined region of the reference image, and extracts a value having a frequency equal to or higher than a predetermined threshold as a feature amount. The pixel component for generating the histogram is the same as the pixel component for calculating the correlation degree by the correlation degree calculation unit 113, for example. Specific examples of pixel components include hue (H) in an HSV image represented by HSV representation of hue (H), saturation (S), and brightness (V). Note that there are various methods for extracting feature amounts, and the processing example of this embodiment is merely an example.

  The feature matching degree determination unit 117 determines the degree to which the input image supplied from the image processing unit 106 matches the feature amount extracted by the feature amount extraction unit 116. The degree of matching with the feature amount is a ratio of the number of pixels determined to match the feature amount extracted by the feature amount extraction unit 116 to the number of pixels of the entire input image. Note that there are various methods for determining the degree of feature matching, and the processing example of this embodiment is merely an example.

  The subject extraction unit 118 determines that the reference region having the maximum evaluation value calculated by the evaluation value calculation unit 115 is the subject region, and extracts an image of the subject region. By determining the subject area from the sequentially supplied input images, the target subject can be tracked. The extracted image of the subject area is supplied to the reference image registration unit 111 and replaces the registered reference image. Information on the determined subject area is given to the control unit 105.

  The control unit 105 performs automatic focus detection (AF) control and automatic exposure (AE) control in the imaging apparatus 100. Specifically, the control unit 105 controls the focus control mechanism and the exposure control mechanism (neither shown) of the lens unit 101 based on the image output from the A / D conversion unit 104. For example, the focus control mechanism is an actuator that drives the lens unit 101 in the optical axis direction, and the exposure control mechanism is an actuator that drives an aperture or a shutter (when there is a mechanical shutter).

  The control unit 105 can use the extraction result of the subject area supplied from the subject tracking unit 110 in the determination of the imaging condition by the AF control or the AE control. Specifically, the control unit 105 performs AF control by controlling the focus detection condition so that the region extracted as the subject region is in focus, or controls the exposure condition so that the exposure of the region is appropriate. Thus, AE control can be performed. The control unit 105 also performs readout control of the image sensor 102 such as control of output timing of the image sensor 102 and selection control of pixels to be read out.

  Note that when the imaging apparatus 100 has a configuration for realizing a function of detecting a specific subject, such as a face detection unit, the detected subject region (for example, a face region) can be automatically registered as a reference image. Also, a predetermined area (focus detection area, photometry area, etc.) in the image used for AF control or AE control may be registered in the reference image as an initial subject area. In these cases, the imaging apparatus 100 may not have the subject specifying unit 109.

  Here, a method of calculating the evaluation value in the evaluation value calculation unit 115 will be described. In this embodiment, the evaluation value calculation unit 115 calculates an evaluation value for each reference region as a function value of the correlation degree obtained by the correlation degree calculation unit 113 and the distance obtained by the distance calculation unit 114.

  As described above, when the evaluation value is obtained based only on the degree of correlation, if there is a region (similar region) similar in characteristics to the subject region (reference image) in addition to the true subject region, the similar region is determined as the subject region. May be erroneously detected. In particular, if the appearance of the subject changes between when the subject region used as the reference image is extracted and when the input image to be tracked is captured, the similar region is more likely than the true subject region. In some cases, the degree of correlation with the reference image increases. That is, when there is a high possibility that a similar region exists, the reliability of the correlation degree obtained by the correlation degree calculation unit 113 is reduced.

  For this reason, in the present embodiment, the feature coincidence determination unit 117 calculates the coincidence as an index of the possibility of the presence of the similar region, and if it is determined that the similar region is likely to exist, the evaluation value Increase the degree to which distance contributes. This makes it easy to determine the subject area from the reference area with a small distance from the nearest subject area, and reduces the possibility of erroneously detecting a similar area with a high degree of correlation in a general situation where the movement of the subject is not large. be able to. On the other hand, when it is determined that the possibility that a similar region exists is small, the reliability of the correlation calculated by the correlation calculation unit 113 is high. Therefore, the extent to which the distance contributes to the evaluation value is lowered. As a result, even if the distance from the nearest subject area is large, a reference area having a high degree of correlation is easily determined as the subject area, and the subject area can be accurately detected even when the subject moves greatly.

  As described above, in the present embodiment, the evaluation value of the reference region is calculated in consideration of both the degree of correlation and the distance from the nearest subject region, and the nearest subject region is determined according to the possibility of the presence of a similar region. The degree of considering the distance from is dynamically changed. Therefore, it is possible to track a subject that moves greatly while suppressing erroneous detection when a similar region exists.

In the present embodiment, the evaluation value calculation unit 115 calculates an evaluation value for each reference region as a function value of the correlation degree obtained by the correlation degree calculation unit 113 and the distance obtained by the distance calculation unit 114, for example, using the following equation (1). .
Evaluation value = 1 / {(1 / correlation degree) + distance from the latest subject area × α} (1)

  Here, α is a coefficient, and when the degree of coincidence is high (that is, the possibility that a similar area exists is high), a large value is low, and the degree of coincidence is low (that is, the possibility that a similar area exists) is low. In this case, a small value is set by the evaluation value calculation unit 115. The specific relationship between the degree of coincidence and the coefficient value can be experimentally obtained in advance, and the evaluation value calculation unit 115 may have a table in which the degree of coincidence is associated with the coefficient value. , A relational expression that can obtain a coefficient value by substituting the degree of coincidence may be provided. The subject extraction unit 118 determines the reference region having the highest evaluation value as the subject region.

The reference image (the image of the nearest subject area) and the reference area are rectangular areas with a size of horizontal M pixels × vertical N pixels, the position of the nearest subject area I is (x0, y0), and the position of the reference area is ( x1, y1). As described above, the position of the region can be determined in advance, such as the position of the center of gravity or the position of the vertex. Further, when the degree of correlation between the standard image and the reference region image is obtained as the magnitude of the absolute value of the difference sum of the pixel values at the corresponding positions, the equation (1) for obtaining the evaluation value SAD is the following equation (2): become that way.

  The first term on the right side of Equation (2) represents the degree of correlation (the smaller the value, the higher the degree of correlation), and the second term represents the distance (the smaller the value, the shorter the distance). Here, since the evaluation is higher when the SAD is larger, the reciprocal of the right side is obtained, but the evaluation value may be higher as the value obtained by the mathematical expression in the parentheses on the right side is smaller.

  In the present embodiment, the feature amount extraction unit 116 and the feature matching degree determination unit 117 obtain a matching degree as a value indicating the possibility of the presence of a similar region (or the reliability of the reference image), and evaluate according to the matching degree. A configuration has been described in which the degree to which the distance contributes to the value is changed. However, this is merely an example, and other arbitrary configurations may be used as long as similar values can be obtained. For example, the average value and the maximum value of the correlation degree calculated by the correlation degree calculation unit 113 are calculated for each reference region, and the value obtained by dividing the maximum value of the correlation degree by the average value of the correlation degree or the absolute value of the difference is matched. It may be used in the same way as the degree. Here, it is assumed that the higher the degree of correlation, the larger the degree of correlation. When there are many regions similar to the reference image in the input image, the difference between the maximum value of the correlation degree and the average value of the correlation degree becomes small, so the value of the maximum value / average value or | maximum value−average value | Get smaller. On the other hand, when the number of similar regions is small, the difference between the maximum value of the correlation degree and the average value of the correlation degree is large, so that the value of the maximum value / average value or | maximum value−average value |

  When using the maximum value and the average value of the correlation degree, the evaluation value calculation unit 115 can calculate the value indicating the possibility of the presence of the similar region or the reliability of the reference image, and the feature amount extraction unit 116 and the feature matching degree The determination unit 117 is not necessary.

(Operation of subject tracking unit 110)
Next, details of the processing of the subject tracking unit 110 will be described with reference to FIGS. In the following description, it is assumed that a specific subject to be detected and tracked is a human face.

  In step S <b> 201, the reference image registration unit 111 reads captured images sequentially captured by the image sensor 102 as input images. Note that the input image reading in S201 may be performed for all captured frames or may be performed intermittently.

  Next, in step S <b> 202, the reference image registration unit 111 registers an initial reference image (reference image used first) from the input image based on the partial area specified by the subject specifying unit 109. FIG. 3A shows an example of the reference image 301 registered based on the designation from the subject designation unit 109. When the subject tracking is started, the reference image 301 becomes the subject extraction result of the subject extraction unit 118.

  When the next input image is read in S <b> 203, the reference image registration unit 111 gives the reference image to the feature amount extraction unit 116. In step S204, the feature amount extraction unit 116 extracts feature amounts from the reference image. For example, the feature amount extraction unit 116 generates a histogram for predetermined component values of pixels included in the reference image, and extracts a value having a frequency equal to or higher than a predetermined threshold value as the feature amount. The pixel component for generating the histogram is the same as the pixel component for calculating the correlation degree by the correlation degree calculation unit 113, for example. Specific examples of pixel components include hue (H) in an HSV image represented by HSV representation of hue (H), saturation (S), and brightness (V). Note that there are various methods for extracting feature amounts, and the processing example of this embodiment is merely an example. FIG. 3A shows an example of a hue histogram 302 generated from each pixel of the reference image 301. The feature amount extraction unit 116 extracts a range of hue values indicating a frequency equal to or higher than a predetermined threshold in the hue histogram 302 as a feature amount.

  In S205, the feature matching degree determination unit 117 matches the feature amount of the reference image extracted by the feature amount extraction unit 116 among the pixels included in the input image (the hue value is included in the feature amount range). Are counted and the ratio to all pixels is calculated. For example, the feature matching degree determination unit 117 generates a binarized image from the input image in which the pixel that matches the feature amount is a white pixel and the non-matching pixel is a black pixel, and the ratio of the number of white pixels to the total number of pixels is It is calculated as the degree that the input image matches the feature amount. FIG. 3B shows an example of the input image 303 at time t = n and an image 304 obtained by binarizing the input image 303 with the feature amount obtained from the hue histogram 302 of FIG. Furthermore, an example of an input image 305 at time t = m (m> n) and an image 306 obtained by binarizing the input image 305 with the feature amount obtained from the hue histogram 302 of FIG.

  In S206, the evaluation value calculation unit 115 determines the degree of contribution of the distance to the evaluation value (value of the coefficient α) according to the degree of matching calculated by the feature matching degree determination unit 117. As described above, when the feature matching degree is low (feature pixel ratio is small) as in the feature binarized image 304 shown in FIG. 3B, in order to reduce the degree that the distance contributes to the evaluation value, The value of the coefficient α is also reduced. Further, when the degree of coincidence is large as in the feature binarized image 306, the value of the coefficient α is also increased in order to increase the degree that the distance contributes to the evaluation value.

  In S207, the reference area setting unit 112 sets a reference area corresponding to the size of the standard image for the input image. In S208, the correlation degree calculation unit 113 calculates the degree of correlation between the reference area image and the standard image from each pixel of the reference area image and the standard image. In S209, the distance calculation unit 114 calculates the distance between the position of the reference area for which the degree of correlation is obtained and the standard position. The reference position is the position of the most recently determined subject area (that is, the position of the reference image in the image from which the reference image is extracted).

  In S210, the evaluation value calculation unit 115 uses the correlation degree calculated by the correlation calculation unit 113, the distance calculated by the distance calculation unit 114, and the coefficient α determined in S206, based on Expression (2). Is calculated. The degree of correlation may be substituted into the first term on the right side of equation (2), and the distance may be substituted into the second term. In S211, the evaluation value calculation unit 115 determines whether evaluation values have been calculated for all reference areas to be set in the entire input image or in a predetermined search area.

  When the reference area to be set still remains (S211: NO), in S207, the reference area setting unit 112 newly sets a reference area whose position is shifted by a predetermined number of pixels in a predetermined direction, and creates a new reference area. On the other hand, the processing of S208 to 210 described above is performed.

  When the evaluation value is calculated for all the reference areas to be set (S211: YES), the evaluation value calculation unit 115 outputs information on the reference area having the maximum evaluation value to the subject extraction unit 118. . In step S212, the subject extraction unit 118 determines and extracts an image corresponding to the reference region having the maximum evaluation value from the input images as the subject region. The subject extraction unit outputs the extracted image to the reference image registration unit 111. Also, information regarding the determined subject area is output to the control unit 105, the image processing unit 106, and the distance calculation unit 114.

  In step S <b> 213, the reference image registration unit 111 updates the reference image based on the subject area extracted by the subject extraction unit 118. The updated reference image is used in subject tracking processing (S203 to S212) of the input image at the next time. In this way, by sequentially updating the reference image based on the extracted subject area, even when the subject appearance changes in time series, such as when the orientation of the subject changes, subject tracking is performed appropriately. be able to. By updating the reference image, the feature amount in the reference image obtained in S204 is also updated. On the other hand, when the change in appearance of the subject in time series is not taken into consideration, the reference image registered in the initial stage may be maintained without updating the reference image. In this case, the feature amount in the reference image is also maintained.

  In the present embodiment, for ease of explanation and understanding, the processing steps of the subject tracking processing are described as being executed in series. However, processing steps that can be processed in parallel may be executed simultaneously. Needless to say. For example, a feature matching degree is obtained for the input image and a coefficient α is determined (S204 to S206), and a reference area is set for the input image and a correlation degree and a distance are calculated (S207 to 209). May be processed in parallel.

  As described above, according to the present embodiment, in the subject tracking device that tracks a subject using the reference image representing the subject to be tracked and the correlation between the input image, the latest determination is made in addition to the correlation. An evaluation value in consideration of the distance from the subject area is calculated for each reference area. Furthermore, when the input image includes a large proportion of pixels having the same characteristics as the base image, the proportion of the distance that contributes to the evaluation value is increased so that the reference region with a short distance is easily determined as the subject region. . In addition, when the ratio of pixels having the same characteristics as the base image is small in the input image, the ratio of the distance to the evaluation value is reduced, and the reference area having a high degree of correlation is easily determined as the subject area. To do. Therefore, it is possible to suppress false detection when the reliability of the correlation degree is low, and to detect the subject accurately even when the movement of the subject is large when the reliability of the correlation degree is high, Stable subject tracking is possible.

(Other embodiments)
In the above-described embodiment, the imaging apparatus as an example of the subject tracking apparatus has been described. However, as described above, the present invention can be applied to various devices other than the imaging apparatus. For example, when it is applied to an image data reproduction display device, it can be applied to set image data reproduction conditions and display conditions using information on the subject area in the image data (position, size, etc. of the subject in the image). It is. Specifically, information indicating the subject such as a frame is superimposed on the subject position in the image, and brightness and color tone are adjusted so that the subject portion is appropriately displayed according to the luminance and color information of the subject portion. Display conditions can be controlled.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

  In addition, although the functional block in a figure is realizable with a hardware, software, or those combination, a functional block and the structure which implement | achieves it do not need to respond | correspond 1: 1. A plurality of functional blocks may be realized by a single software or hardware module.

Claims (7)

A subject tracking device that tracks a specific subject in an image over a plurality of input images input in time series,
Correlation degree calculating means for obtaining a correlation degree between the reference image registered in the registration means and each of the plurality of reference regions set in the input image;
Extracting means for extracting a feature amount of the reference image;
A degree of coincidence determination means for determining the degree to which the input image matches the feature amount;
Distance calculating means for calculating a distance between each of the plurality of reference regions and a predetermined reference position in the input image;
For each of the plurality of reference regions, an evaluation value calculation unit that calculates an evaluation value using the degree of correlation, the distance, and the degree determined by the matching degree determination unit;
Determination means for determining that the reference area having the largest evaluation value is the area of the specific subject among the plurality of reference areas;
The predetermined reference position is the position of the area of the specific subject that has been most recently determined by the determination unit,
The evaluation value calculation means calculates the evaluation value as a function value of the correlation degree and the distance, and the degree to which the distance contributes to the evaluation value is determined by the degree of coincidence determination means. An object tracking apparatus, wherein the evaluation value is calculated by dynamically changing the value so that the evaluation value is larger as it is higher and smaller as the degree of determination by the matching degree determination unit is lower.   2. The extraction unit according to claim 1, wherein the extraction unit generates a histogram for a predetermined component value of a pixel included in the reference image, and extracts a value having a frequency equal to or higher than a predetermined threshold as the feature amount. The subject tracking device described.   The coincidence degree determination unit counts pixels that match the feature amount among pixels included in the input image, and calculates a ratio of the number of pixels that matches the feature amount to the number of pixels of the entire input image. The subject tracking device according to claim 1, wherein the subject tracking amount is calculated as a degree of coincidence with the feature amount.   4. The image processing apparatus according to claim 1, further comprising an update unit configured to update the reference image with an image of the input image corresponding to the specific subject area determined by the determination unit. The subject tracking device according to Item. A designating unit for designating a region in the input image;
The subject tracking device according to claim 1, wherein the reference image used first is an image of an area designated by a user through the designation unit. A method for controlling a subject tracking device for tracking a specific subject in an image over a plurality of input images input in time series,
Correlation degree calculating means for obtaining a correlation degree between a reference image registered in the registration means and each of a plurality of reference regions set in the input image;
An extracting step in which an extracting means extracts a feature amount of the reference image;
A degree of coincidence determination means for determining a degree of coincidence of the input image with the feature amount;
A distance calculating step of calculating a distance between each of the plurality of reference regions and a predetermined reference position in the input image;
An evaluation value calculating unit that calculates an evaluation value for each of the plurality of reference regions using the correlation, the distance, and the degree determined in the coincidence determination step;
A determination unit that determines a reference region having the largest evaluation value among the plurality of reference regions as a region of the specific subject;
The predetermined reference position is a position of the area of the specific subject that has been most recently determined by the determination step;
The evaluation value calculating step calculates the evaluation value as a value of a function of the correlation degree and the distance, and the degree to which the degree that the distance contributes to the evaluation value is determined in the coincidence degree determining step A method for controlling a subject tracking apparatus, wherein the evaluation value is calculated by dynamically changing so that the evaluation value is dynamically increased so as to increase as the value increases and decreases as the degree determined in the matching degree determination step decreases.   The program for making a computer perform each process of the control method of the to-be-photographed object tracking apparatus of Claim 6.

Images