JP7749372B2 - Image processing device and control method thereof, imaging device, and program - Google Patents

Description

The present invention relates to display control based on image evaluation values.

Image processing performed within an imaging device includes detecting the subject area contained in the captured image, and the subject to be targeted for autofocus (AF). Image display devices such as LCD displays and electronic viewfinders display the subject detection results to visually present the focus area (AF frame, etc.) to the user. Patent Document 1 discloses technology that acquires a video signal from a tracking image sensor via a signal processing unit and detects the movement of the AF target subject based on the video signal.

Japanese Patent Application Laid-Open No. 2005-338352

With conventional technology, when the subject is a person, it is difficult to identify the ranging position within the subject area when the size of the detected subject is relatively large, such as when the person's eyes cannot be detected from the image data and the entire body is detected, etc. On the display screen of the subject detection results, it may be difficult to visually determine which area within the subject area is set as the ranging position.
An object of the present invention is to provide an image processing device capable of display control based on an evaluation value obtained from a detected image of a subject.

An image processing device according to one embodiment of the present invention comprises a detection means for detecting a first region in an image corresponding to a subject and a second region in the image corresponding to a part of the subject ; a calculation means for calculating an evaluation value for each of a plurality of feature points in the image from information on the feature points ; and a control means for controlling the output of information regarding the ranging region corresponding to the ranging position in the image to a display means displaying the image based on the evaluation value, wherein the control means controls the display means to display information indicating the first region, and to display a plurality of pieces of information indicating regions corresponding to the plurality of evaluation values as candidates for the ranging region corresponding to the second region in the display region in the image corresponding to the first region, and controls the number of the plurality of pieces of information by controlling a threshold value for the evaluation value.

The image processing device of the present invention enables display control based on evaluation values obtained from detected images of a subject.

1 is a block diagram showing a configuration of an image processing apparatus according to a first embodiment. FIG. 2 is a block diagram showing an image processing unit in the first embodiment. 10 is a flowchart showing evaluation value display control in the first embodiment. 10 is a flowchart illustrating a feature point detection process according to the first embodiment. 10 is a flowchart illustrating a similarity calculation process according to the first embodiment. FIG. 10 is a schematic diagram of a feature calculation process in similarity calculation. 10 is a flowchart illustrating a reliability determination process according to the first embodiment. 10 is a table showing the relationship between evaluation values and each element in the first embodiment. 10 is a flowchart illustrating control of the number of frames to be displayed in the first embodiment. FIG. 10 is a schematic diagram of a frame display based on an evaluation value in the first embodiment. 10 is a flowchart illustrating frame display control according to density in the first embodiment. 5A and 5B are schematic diagrams illustrating frame display control according to density in the first embodiment. 10 is a flowchart showing evaluation value display control in the second embodiment. FIG. 11 is a schematic diagram illustrating a color-coded display of evaluation values in the second embodiment. 10 is a flowchart illustrating a ranging area control in the second embodiment. 10A and 10B are schematic diagrams illustrating an example of distance measurement area control in the second embodiment. 10A and 10B are schematic diagrams showing another example of distance measurement area control in the second embodiment. 11 is a flowchart showing evaluation value display control in the third embodiment. 13A to 13C are schematic diagrams illustrating differences in subject detection size in the third embodiment. 13 is a flowchart showing evaluation value display control in the fourth embodiment. FIG. 13 is a schematic diagram of an evaluation value display in the fourth embodiment. 13 is a flowchart showing a tracking process in the fifth embodiment. FIG. 13 is a diagram showing a search range for tracking a ranging frame in the fifth embodiment. 13 is a flowchart showing display control of a range finding frame in the fifth embodiment.

Embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Examples are given in which an image processing device is applied to an imaging device such as a digital still camera or digital video camera.

(First embodiment)
1 is a block diagram showing an example of the configuration of an image processing device 100 according to this embodiment. An imaging device to which the image processing device 100 is applied includes an imaging optical unit 101 and an image sensor 102. The imaging optical unit 101 includes optical components such as lenses and apertures that constitute an imaging optical system, a drive mechanism, and a drive circuit. The drive circuit of the imaging optical unit 101 is electrically connected to a bus 116. The image sensor 102 is a CCD (charge-coupled device) image sensor or a CMOS (complementary metal-oxide semiconductor) image sensor, which performs photoelectric conversion on a formed subject image and outputs an electrical signal corresponding to the subject image.

The A/D conversion unit 103 acquires the analog image signal output by the image sensor 102 and converts it into a digital image signal. The image processing unit 104 acquires the digital image signal from the A/D conversion unit 103 and performs various image processing. The image processing unit 104 is composed of various processing circuits and buffer memory, etc., and appropriately performs gamma correction, white balance processing, etc. on the A/D converted digital image data.

The data transfer unit 105 is composed of multiple DMACs (Direct Memory Access Controllers) that transfer data. The bus 116 is a system bus for transmitting control signals from the CPU (Central Processing Unit) 114 (described below) and other devices. The bus 117 is a data bus for transferring image data and the like. The data transfer unit 105 is connected to the image processing unit 104 and the bus 117.

The memory control unit 106 controls the DRAM (Random Access Memory) 107. The DRAM 107 stores data such as still images, moving images, and audio, as well as constants and programs for the operation of the CPU 114. The memory control unit 106 is electrically connected to the buses 116 and 117, and writes data to and reads data from the DRAM 107 in accordance with instructions from the CPU 114 or the data transfer unit 105.

The non-volatile memory control unit 108 controls the ROM (Read Only Memory) 109. The ROM 109 is electrically erasable and recordable, and may be an EEPROM (Electrically Erasable Programmable Read-Only Memory). The ROM 109 stores constants, programs, etc. for the operation of the CPU 114. The non-volatile memory control unit 108 is electrically connected to the bus 116, and writes data to and reads data from the ROM 109 in accordance with instructions from the CPU 114.

The recording media control unit 110 controls the recording media 111. The recording media 111 is a recording medium such as an SD card. The recording media control unit 110 is electrically connected to buses 116 and 117, and records image data to the recording media 111 and reads the recorded data.

The display control unit 112 controls the display unit 113. The display unit 113 has an LCD display, an electronic viewfinder, etc., and performs display processing of image data transferred from the image processing unit 104, menu screens, etc. When taking still images or videos, the image processing unit 104 processes image data input from the A/D conversion unit 103 in real time, and the display control unit 112 controls the display of the processed data on the display unit 113. The display control unit 112 is electrically connected to buses 116 and 117.

The CPU 114 controls the operation of the image processing device 100 via the bus 116. The CPU 114 executes programs stored in the ROM 109 to realize various functions. The operation unit 115 includes switches, buttons, a touch panel, etc. that are operated by the user, and is used to turn the power and shutter on/off, etc. The operation unit 115 is electrically connected to the bus 116, and transmits operation instruction signals from the user to the CPU 114.

Figure 2 is a block diagram showing an example configuration of the image processing unit 104. The image processing unit 104 includes a feature point detection unit 201, a saliency calculation unit 202, a similarity calculation unit 203, a density calculation unit 204, a reliability determination unit 205, and an evaluation value calculation unit 206. Each unit can refer to the information acquired up to the previous stage as needed.

The feature point detection unit 201 acquires image data from the A/D conversion unit 103 and detects feature points within the image. The saliency calculation unit 202 acquires feature point information from the feature point detection unit 201 and calculates the saliency of the feature points. The similarity calculation unit 203 acquires information from the saliency calculation unit 202 and calculates the similarity between feature points. The density calculation unit 204 acquires information from the similarity calculation unit 203 and calculates the density of feature points within an image area. Density is an index that indicates how densely feature points are located within an image area. The reliability determination unit 205 acquires information from the density calculation unit 204 and performs reliability determination mainly using similarity information. The evaluation value calculation unit 206 calculates an evaluation value based on one or more of the saliency, density, and reliability determination results acquired in the previous stages.

Selective evaluation value display control according to subject detection results will be described in detail with reference to Figure 3. Figure 3 is a flowchart showing the overall processing flow for selective evaluation value display control according to subject detection conditions. In this embodiment, an example of frame display based on evaluation values is shown. Known technologies can be used for subject detection methods and methods for detecting specific subject parts such as the pupils, face, and head. For example, subject detection methods include methods that use features such as the eyes, nose, and mouth, and detection methods that use learning algorithms such as neural networks.

In S301, the CPU 114 executes processing to detect a subject within an image. The subject is, for example, a person or an animal. In S302, the feature point detection unit 201 detects feature points from the image area (subject area) of the subject detected in S301. A specific example of the feature point detection processing will be described later using FIG. 4. In S303, the image processing unit 104 calculates saliency, density, and similarity, and determines reliability, and calculates an evaluation value for the feature points detected in S302. Specific processing will be described later. In S304, the CPU 114 determines whether the subject detected in S301 is a specific subject. An example of a specific subject is a person. If it is determined that the detected subject is a specific subject, processing proceeds to S305. If it is determined that the detected subject is not a specific subject, processing proceeds to S311.

In S305, the CPU 114 determines whether the pupils of the specified subject can be detected. If it is determined that the pupils of the specified subject can be detected, the process proceeds to S308. If it is determined that the pupils of the specified subject cannot be detected, the process proceeds to S306. In S306, the CPU 114 determines whether the face of the specified subject can be detected. If it is determined that the face of the specified subject can be detected, the process proceeds to S309. If it is determined that the face of the specified subject cannot be detected, the process proceeds to S307. In S307, the CPU 114 determines whether the head of the specified subject can be detected. If it is determined that the head of the specified subject can be detected, the process proceeds to S310. If it is determined that the head of the specified subject cannot be detected, the process proceeds to S311.

In S308, S309, S310, and S311, the CPU 114 performs control to display a frame in the detected area. That is, in S308, the display unit 113 displays an pupil frame for the detected pupil area, and in S309, the display unit 113 displays a face frame for the detected face area. In S310, the display unit 113 displays a head frame for the detected head area, and in S311, the display unit 113 displays a subject frame for the detected subject area.

After S308, S309, and S310, the process proceeds to S314, and after S311, the process proceeds to S312. In S312, the CPU 114 enables frame display according to the evaluation value calculated in S303 for a subject that was determined not to be a specific subject in S304 or for a subject for which it was determined that no part of the subject (eyes, face, head) could be detected in S305 to S307. In the next step, S313, a selection process for evaluation value display (frame display) is performed, and then the process proceeds to S314. Details of evaluation value display will be described later.

In S314, the CPU 114 determines the ranging area (focus state detection area) based on the frame display selected in S313. In S315, the CPU 114 controls the display unit 113 to display the ranging area determined in S314. This completes the series of processes.

The processing in Figure 3 will be explained in detail below using Figures 4 to 12. Figure 4 is a flowchart of the processing performed by the feature point detection unit 201 and the saliency calculation unit 202. In S401, image data of the subject area detected in S301 of Figure 3 is acquired. In S402, the feature point detection unit 201 specifies an area for which feature point detection processing is to be performed on the image data of the detected subject area.

In S403, the feature point detection unit 201 performs horizontal first derivative filter processing on the area specified in S402, generating a horizontal first derivative image. In S404, the feature point detection unit 201 further performs horizontal first derivative filter processing on the horizontal first derivative image acquired in S403, generating a horizontal second derivative image.

The processes of S405 and S406 are vertical differential filter processes of the image, executed in parallel with S403 and S404, respectively. In S405, the feature point detection unit 201 performs vertical first differential filter processing on the area specified in S402, generating a vertical first differential image. In S406, the feature point detection unit 201 further performs vertical first differential filter processing on the vertical first differential image acquired in S405, generating a vertical second differential image.

The processing of S407 is image differential filtering performed as parallel processing to S404. In S407, the feature point detection unit 201 further performs vertical first differential filtering on the horizontal first differential image acquired in S403, generating horizontal first differential and vertical first differential images.

After S404, S406, and S407, the process proceeds to S408. In S408, the saliency calculation unit 202 calculates the determinant (denoted as Det) of the Hessian matrix (denoted as H) of the differential values acquired in S404, S406, and S407. The horizontal second-order differential value acquired in S404 is denoted as L _xx , the vertical second-order differential value acquired in S406 is denoted as L _yy , and the horizontal first-order differential and vertical first-order differential values acquired in S407 are denoted as L _xy . The Hessian matrix H is expressed by the following formula (1), and the determinant Det is expressed by the following formula (2).

In S409, the saliency calculation unit 202 determines whether the value of the determinant Det calculated in S408 is greater than or equal to zero. If it is determined that the value of the determinant Det is greater than or equal to zero, the process proceeds to S410. If it is determined that the value of the determinant Det is less than zero, the process proceeds to S411.

In S410, the feature point detection unit 201 performs processing to detect points where the value of the determinant Det is zero or greater as feature points, and then proceeds to processing in S411. In S411, the feature point detection unit 201 or the saliency calculation unit 202 determines whether processing has been performed for all of the subject regions input in S401. If it is determined that processing has been completed for all of the target regions, the feature point detection processing in Figure 4 ends. On the other hand, if there are unprocessed target regions, the processing returns to S402, and the processing from S402 to S410 is repeated.

Figure 5 is a flowchart of the processing performed by the similarity calculation unit 203. In S501, the similarity calculation unit 203 calculates feature amounts for the feature points detected by the feature point detection unit 201 in S410 of Figure 4. Figure 6 is a schematic diagram illustrating the feature amount calculation process. A point of interest 601 in the image is indicated by a black dot. A number of random line segment patterns 602 are shown around the feature point of interest 601. In this embodiment, an example is shown in which the magnitude relationship between the luminance values at both ends of each line segment is expressed as 1s and 0s. The magnitude relationship between all 1s and 0s for the line segment pattern 602 is expressed as a bit string, and the quantity is calculated as the feature amount.

In S502 of FIG. 5, a determination process is performed to determine whether feature calculation has been completed for all feature points detected in S410 of FIG. 4. If it is determined that feature calculation has been completed for all feature points, the process proceeds to S503. If it is determined that feature calculation has not been completed, the feature calculation process in S501 is repeated.

In S503, the similarity calculation unit 203 specifies a feature point for which similarity calculation processing is performed, and then in S504, specifies a counterpart feature point for comparing similarity with the feature point specified in S503. In S505, the similarity calculation unit 203 calculates the Hamming distance (denoted as D) of the feature amounts between the feature points specified in S503 and S504. The bit string of the feature amount of the feature point specified in S503 is denoted as A, and its elements are denoted as Ai. The bit string of the feature amount of the feature point specified in S504 is denoted as B, and its elements are denoted as Bi. The Hamming distance D, which represents similarity, is expressed by the following formula (3).
In equation (3), xor represents an exclusive OR operation.

In S506, a determination is made as to whether the calculation of the Hamming distance D between the feature point focused on in S503 and all feature points has been completed. If it is determined that the calculation of the similarity (Hamming distance) has been completed, the process proceeds to S507. If it is determined that the calculation of the similarity (Hamming distance) has not been completed, the process returns to S504, and the processes of S504 and S505 are repeatedly executed.

In S507, a determination is made as to whether processing has been completed for all feature points. If it is determined that processing has been completed for all feature points, the similarity calculation process ends. If it is determined that processing has not been completed for all feature points, the process returns to S503, and the processes from S503 to S507 are repeated.

Figure 7 is a flowchart of the processing performed by the reliability determination unit 205. In S701, the reliability determination unit 205 specifies a feature point to focus on in the reliability determination, and in S702, specifies a feature point to be compared with the feature point specified in S701. In S703, the reliability determination unit 205 compares the similarity between the feature points specified in S701 and S702 with a threshold value (denoted as M) based on the similarity calculated by the similarity calculation unit 203. A determination process is performed to determine whether the similarity is greater than or equal to the threshold value (M or greater). For example, if the number of bits in the feature bit string is 50, the similarity is maximized when the Hamming distance D is 0. The threshold value M is set to 10, for example. However, the number of bits in the bit string and the threshold value M are variable. If it is determined that the similarity is greater than or equal to the threshold value, the process proceeds to S704. If it is determined that the similarity is less than the threshold value, the process proceeds to S705.

In S704, the reliability determination unit 205 performs a repetitive pattern determination for the feature points whose similarity was determined to be equal to or greater than the threshold M in S703. A repetitive pattern is a situation in which there are multiple similar features within the same image, and indicates a feature with a high possibility of false detection and low reliability. For example, this could be a shooting scene in which windows of the same shape are arranged consecutively in a building.

In S705, the reliability determination unit 205 determines whether processing has been completed for all feature points for the feature point focused on in S701. If it is determined that the reliability determination processing for the feature point focused on has been completed, the process proceeds to S706. If it is determined that the reliability determination processing for the feature point focused on has not been completed, the process returns to S702 and continues.

In S706, the reliability determination unit 205 determines whether processing has been completed for all feature points. If it is determined that processing has been completed for all feature points, the reliability determination process ends. If it is determined that the reliability determination process has not been completed, the process returns to S701 and continues the reliability determination process.

Figure 8 is a table showing an example of the evaluation value elements used in the evaluation value calculation by the evaluation value calculation unit 206. The evaluation value calculation unit 206 determines the evaluation value based on the results obtained by the feature point detection unit 201, saliency calculation unit 202, density calculation unit 204, and reliability determination unit 205. Figure 8 shows an example of the relationship between "high" and "low" evaluation values and the corresponding "high" and "low" saliency, density, and reliability. For example, if any of the saliency, density, or reliability is "high," an evaluation value of "high" is obtained.

Based on the evaluation value calculated in S303 of Figure 3, in S312, processing is executed to enable frame display according to the evaluation value for subjects that are not determined to be specific subjects or for subjects in which eyes, faces, or heads are not detected. Hereinafter, the display frame according to the evaluation value will be referred to as the ranging position candidate frame.

Figure 9 is a flowchart explaining display control when the display of distance measurement position candidate frames is enabled. Figure 10 is a schematic diagram showing display examples of the display unit 113 according to the subject detection situation. Figure 10(A) shows a display frame 1001 when the subject's pupil is detected, and Figure 10(B) shows a display frame 1002 when the subject's face is detected. Figure 10(C) shows a display frame 1003 when the subject's head is detected, and Figure 10(D) shows a display frame 1004 when the subject (whole body) is detected. The display example in Figure 10(E) shows a case where the display of distance measurement position candidate frames is enabled, and multiple distance measurement position candidate frames 1005 are displayed.

In S901 of FIG. 9, the image processing unit 104 determines whether the saliency of the feature point calculated by the feature point detection unit 201 and the saliency calculation unit 202 is equal to or greater than a threshold value (denoted as N). As a specific example, the value of the determinant Det of the Hessian matrix H corresponds to the saliency, and the threshold value N is set to 0.5. The threshold value N is a fixed value or a variable value. If the threshold value N is set to a variable value, the number of ranging position candidate frames displayed can be controlled by changing the threshold value N. If it is determined that the value (evaluation value) of the determinant Det is equal to or greater than the threshold value (N or greater), the process proceeds to S902. If it is determined that the value of the determinant Det is less than the threshold value N, the process proceeds to S903.

In S902, the display unit 113 displays the distance measurement position candidate frames. In the next step S903, a determination is made as to whether the display processing of the distance measurement position candidate frames has been completed for all evaluation values. If it is determined that the processing has been completed for all evaluation values, the display processing of the distance measurement position candidate frames ends (proceeds to S313 in Figure 3). On the other hand, if it is determined that the processing has not been completed, the process returns to S901, and the processing from S901 to S903 is repeated.

Figure 11 is a flowchart explaining frame display control according to density when the ranging position candidate frame display is enabled. Figure 12 is a schematic diagram showing display examples on the display unit 113 with and without frame display control according to density. Figure 12(A) shows a display example 1201 when the frame display control of Figure 11 is not performed, in which ranging position candidate frames not based on evaluation values (density, prominence) are comprehensively displayed. If there are a large number of ranging position candidate frames displayed, it may be difficult for the user to select one. In contrast, Figure 12(B) shows a display example 1202 when the frame display control of Figure 11 is performed. Ranging position candidate frames are displayed based on evaluation values (density, prominence).

In S1101 of FIG. 11, the density calculation unit 204 specifies an area for which density is to be calculated. In this embodiment, an example is shown in which the subject area is divided into multiple blocks and density is calculated for each block. In S1102, the density calculation unit 204 calculates the density of feature points in the area specified in S1101. In this embodiment, the density is calculated by counting the number of feature points in the specified area.

In S1103, a determination is made as to whether the density calculated in S1102 is equal to or greater than a threshold value (denoted as P). The threshold value P is a fixed value or a variable value. If the threshold value P is a variable value, the number of ranging position candidate frames displayed can be controlled by controlling the threshold value P. If it is determined in S1103 that the density is equal to or greater than the threshold value (P or greater), the process proceeds to S1104. If it is determined that the density is less than the threshold value P, the process proceeds to S1106.

In S1104, the density calculation unit 204 specifies feature points whose saliency is higher than the threshold value in the areas where the density is determined to be equal to or higher than the threshold value P in S1103. In S1105, the display unit 113 displays a ranging position candidate frame centered on the feature point specified in S1104. This makes it possible to display a representative frame with a high evaluation value for areas with high density. Furthermore, in S1106, the display unit 113 displays a ranging position candidate frame according to saliency for areas where the density is determined to be less than the threshold value P in S1103.

Following S1105 and S1106, in S1107, a determination is made as to whether processing has been completed for the entire target area. If it is determined that processing has been completed for the entire target area, frame display control according to density is terminated. On the other hand, if it is determined that processing has not been completed for the entire target area, the process returns to S1101, and the processes from S1101 to S1106 are repeated.

In S313 of Figure 3, a process is performed to select an arbitrary frame from the displayed candidate frames for ranging positions. An arbitrary frame is selected from the candidate frames for ranging positions displayed as in the example of Figure 10 (E) or Figure 12 (B). There are automatic selection processes based on predetermined conditions, manual selection processes, and semi-automatic selection processes based on the presentation of recommended options. Here, an example of frame selection by user operation is shown. In an embodiment equipped with a touch panel, the user can specify the selection by touching the frame they want to select as the ranging position from among multiple frames. In addition to touch operation, button operation, stick operation, etc. may also be used as a selection operation method.

In this embodiment, an evaluation value is calculated using the saliency, density, and reliability of feature points detected in the image, and evaluation value display control, for example, display control of candidate ranging position frames, is performed according to the subject detection situation. This allows the user to visually recognize candidate ranging positions by displaying frames even when the eyes, face, or head of a specific subject cannot be detected. Furthermore, by controlling the threshold for displaying candidate ranging position frames, it is possible to display area frames with relatively high evaluation values. By allowing the user to select the ranging area they intend, it is possible to determine a ranging area that balances ranging performance with the user's intentions.

(Modification of the first embodiment)
In the first embodiment, a saliency calculation method based on feature point detection using a Hessian matrix has been described, but other calculation methods such as edge detection or corner detection can also be used. Furthermore, with regard to calculation of feature amounts of feature points, a calculation method based on the magnitude relationship between the luminance values of the feature points and their surroundings has been described, but a feature amount calculation method based on hue or saturation can also be used. Furthermore, with regard to frame display control according to density, a method has been described in which a representative ranging position candidate frame is displayed by specifying an area with high saliency, but a representative ranging position candidate frame can also be determined using the center of gravity of the area, etc.

In the first embodiment, no mention is made of the color or line type of the display frame, but in the modified example, processing is performed to change the color or line type of the display frame depending on the calculated evaluation value. For example, the display unit 113 displays a first display frame for the entire subject in a first color or line type, and a second display frame for a portion of the subject in a second color or line type depending on the evaluation value of each portion, making it easier for the user to see.

Second Embodiment
Next, a second embodiment of the present invention will be described. In the first embodiment, an example of frame display was shown as a display method when evaluation value display control is performed in accordance with the subject detection status. In contrast, in this embodiment, an example of color-coded display is shown as a method of displaying evaluation values in accordance with the subject detection status. Note that detailed description of matters and configurations similar to those in the first embodiment will be omitted, and the description will focus on differences from the first embodiment. This method of omitting description will be the same in the embodiments described below.

Figure 13 is a flowchart explaining display control when color-coded display is enabled, and the display unit 113 is controlled by the CPU 114 and display control unit 112. Figure 14 is a schematic diagram showing an example of color-coded display on the screen of the display unit 113 according to the subject detection status.

In this embodiment, an example is shown in which the subject area is divided into rectangular blocks and the rectangular blocks are displayed in different colors according to the saliency of the feature points within each rectangular block. Within the display area corresponding to the subject area, rectangular blocks with high evaluation values are displayed in a first color, rectangular blocks with medium evaluation values are displayed in a second color, and rectangular blocks with low evaluation values are displayed in a third color. In Figure 14, the color distribution is expressed in shades. That is, rectangular blocks with the first color in the color distribution within the subject area are displayed in the darkest color, and rectangular blocks with the third color are displayed in the lightest color. Note that while an example of a three-level color distribution (high, medium, low) based on evaluation value judgment is shown, this can be expanded to display evaluation values in four or more levels.

In S1301 of FIG. 13, the saliency calculation unit 202 specifies a rectangular block within the subject area to be displayed in a different color. In S1302, the saliency calculation unit 202 determines whether the saliency within the rectangular block specified in S1301 is equal to or greater than a first threshold (denoted as α). If it is determined that the saliency within the rectangular block is equal to or greater than the threshold α, the process proceeds to S1304. If it is determined that the saliency within the rectangular block is less than the threshold α, the process proceeds to S1303.

In S1303, the saliency calculation unit 202 determines whether the saliency within the rectangular block specified in S1301 is greater than or equal to a second threshold (denoted as β, where "α>β"). If it is determined that the saliency within the rectangular block is greater than or equal to the threshold β, the process proceeds to S1305. If it is determined that the saliency within the rectangular block is less than the threshold β, the process proceeds to S1306.

In S1304, the display unit 113 displays the rectangular blocks whose saliency is determined to be equal to or greater than the threshold value α in S1302 in a first color. The density corresponding to the first color is density 1. In S1305, the display unit 113 displays the rectangular blocks whose saliency is determined to be equal to or greater than the threshold value β in a second color. The density corresponding to the second color is density 2. In S1306, the display unit 113 displays the rectangular blocks whose saliency is determined to be less than the threshold value β in S1303 in a third color. The density corresponding to the third color is density 3.

In this embodiment, two threshold levels, α and β, are set, with the relationship "α > β." Three display color levels are set: density 1, density 2, and density 3. Density 1 corresponds to an evaluation value of "high," density 2 corresponds to an evaluation value of "medium," and density 3 corresponds to an evaluation value of "low." The threshold level can be set to any level to change the gradation of the display color.

Following S1304, S1305, and S1306, in S1307, a determination is made as to whether processing has been completed for all target areas. If it is determined that processing has been completed for all target areas, the color-coded display processing according to evaluation values is terminated. If it is determined that processing has not been completed for all target areas, the process returns to S1301, and the processing from S1301 to S1306 is repeated.

The process of determining the ranging area according to the display color will be described with reference to Figure 15. Figure 15 is a flowchart that explains the process in S314 of Figure 3, i.e., the process of determining the ranging area according to the display color of the area selected in S313.

In S1501, the CPU 114 determines whether the density corresponding to the display color of the area selected in S313 of Figure 3 is density 1 or density 2. Here, a determination process is performed to determine whether the evaluation value of the selected area is equal to or greater than a threshold value β. If it is determined that the density corresponding to the display color of the selected area is density 1 or density 2, the process proceeds to S1503. If it is determined that the density corresponding to the display color of the selected area is density 3, the process proceeds to S1502.

In S1502, if the density corresponding to the display color of the selected area is density 3 (evaluation value "low"), the CPU 114 changes the ranging area to include an area with density 1 (evaluation value "high"). Then, the process proceeds to S1503. In S1503, the CPU 114 determines the ranging area according to the display color of the area selected in S313 of Figure 3. After S1503, the process ends.

Figure 16 is a schematic diagram showing differences in ranging areas depending on the selected evaluation value. Figure 16(A) shows the ranging area frame with density 1 (evaluation value "high") selected in S313 of Figure 3. Figure 16(B) shows the ranging area frame determined based on the display color (evaluation value) of the area selected in Figure 16(A). Figure 16(C) shows the ranging area frame with density 3 (evaluation value "low") selected in S313 of Figure 3. Figure 16(D) shows an example in which the ranging area frame has been changed to include an area with a high evaluation value, based on the display color (evaluation value) of the area selected in Figure 16(C). In other words, the ranging area frame includes an area with a "low" evaluation value and multiple surrounding areas with a "high" evaluation value. This example shows an example in which the area used for ranging and subsequent subject tracking processing is changed by internal processing when the selected evaluation value is low.

Figure 17 is a schematic diagram showing an example of displaying and reselecting multiple candidate frames for ranging areas when the density corresponding to the display color of the selected area is density 3 (evaluation value "low"). Figure 17(A) shows the ranging area frame with density 3 (evaluation value "low") selected in S313 of Figure 3. Figures 17(B), (C), and (D) display a ranging position candidate frame that includes the density 3 (evaluation value "low") area and surrounding areas with high evaluation values. The area reselected for the ranging position candidate frame is determined as the ranging area. In Figure 17(B), there is an area with a high evaluation value in the lower right corner of the density 3 (evaluation value "low") area. In Figure 17(C), there is an area with a high evaluation value in the upper right corner of the density 3 (evaluation value "low"). In Figure 17(D), there is an area with a high evaluation value above the density 3 (evaluation value "low") area. In either case, the ranging position candidate frame including areas with relatively high evaluation values is displayed in accordance with the display color (evaluation value) of the ranging area frame shown in Figure 17 (A). The area reselected by user operation is determined as the ranging area.

In this embodiment, the user can visually recognize the candidate ranging positions by displaying them in different colors based on the evaluation value. Furthermore, by controlling the color-coded display using a threshold, areas with relatively high evaluation values are displayed in a distinguishable manner. The user can easily select the desired ranging area, making it possible to determine a ranging area that balances ranging performance with the user's wishes.

(Modification of the second embodiment)
In the second embodiment, a method of displaying ranging position candidate frames by coloring areas divided into rectangular blocks has been described. However, in a modified example, the display areas are not limited to rectangular blocks. For example, a grouping process is performed on areas with similar evaluation values, and ranging position candidate frames are displayed using the grouped areas. Furthermore, instead of color-coded display, icon display may be used so that the user can distinguish between high and low evaluation values.

In a modified example, the number of feature points, their strength, and feature quantity information are used in the evaluation value calculation process. The number of feature points is the number of detected feature points, their density, etc. Feature point strength is an index that represents the strength of features such as cross edges and corners, and the strength decreases when the image area has low contrast, blur, or shaking. Feature quantities are indices that represent the feature points and their surrounding conditions, and tend to have similar values for repeated patterns, etc. If there are a large number of feature points, if the feature point strength is high, or if it is determined from the feature quantities that there are no similar areas, then an evaluation value of "high" is calculated.

(Third embodiment)
Next, a third embodiment of the present invention will be described. In the previous embodiment, an example was shown in which the detection status of a subject was used as a method for controlling the display of an evaluation value. Specifically, display control is performed according to an evaluation value obtained from an image, depending on the detection status of the eyes, face, and head of a specific subject such as a person. In contrast, this embodiment shows an example in which the display of an evaluation value is controlled according to the size of a detected subject, regardless of the type of subject or the specific detection method.

Figure 18 is a flowchart of evaluation value display control according to the size of a detected subject. In S1801, a subject area is detected from a captured image. The subject here is, for example, a human head, a dog, or a flower. In S1802, the feature point detection unit 201 detects feature points from the subject area detected in S1801. The processing in S1802 is the same as S302 in Figure 3.

In S1803, the evaluation value calculation unit 206 calculates the evaluation value of the feature point detected in S1802. The processing of S1803 is the same as that of S303 in FIG. 3. In S1804, the CPU 114 performs processing to determine whether the size of the subject area detected in S1801 is equal to or greater than a threshold value (denoted as Q). If it is determined that the size of the subject area is equal to or greater than the threshold value Q, the processing proceeds to S1805. If it is determined that the size of the subject area is less than the threshold value Q, the processing proceeds to S1809.

In S1805, the CPU 114 performs processing to enable the evaluation value display according to the evaluation value calculated in S1803, and ranging position candidates are displayed. Evaluation value display methods include the frame display method described in the first embodiment and the color-coded display method described in the second embodiment. In S1806, processing is performed to select an arbitrary evaluation value display from the evaluation value displays displayed in S1805. The processing of S1806 is similar to processing of S313 in Figure 3. For example, as a manual selection method, selection is made according to a touch operation, button operation, stick operation, etc. by the user.

In S1807, the CPU 114 performs processing to determine the ranging area based on the evaluation value display selected in S1806. The processing in S1807 is the same as that in S314 of FIG. 3. In S1808, the display unit 113 displays the ranging area determined in S1807. The processing in S1808 is the same as that in S315 of FIG. 3. After S1808, the processing ends.

Figure 19 is a schematic diagram showing differences in the size of detected subject areas. Figure 19(A) shows a situation in which the head 1901 of a subject person has been detected. Because the size of the detected subject area is small, no evaluation value display is performed. Figure 19(B) shows a situation in which the head 1902 of a subject person has been detected, similar to Figure 19(A). Because the size of the detected subject area is large, it is difficult to determine the ranging position within the subject area. In such an example, evaluation value display control is enabled according to the size of the subject area, and ranging position candidates are displayed.

In S1809 of FIG. 18, the display unit 113 displays a subject outline in the detected subject area. If the threshold Q is a variable value, the evaluation value display can be controlled according to the size of the detected subject area by changing the threshold Q. After S1809, the processing ends.

In this embodiment, the evaluation value display is controlled according to the size of the detected subject area (subject size). To address the issue of the ranging area being visually difficult to understand when the subject size is equal to or larger than a certain size, it is possible to visually display ranging position candidates regardless of the subject detection method. Furthermore, by controlling the evaluation value display using a threshold, areas with relatively high evaluation values are displayed in a distinguishable manner. Users can easily select the desired ranging area, making it possible to determine a ranging area that balances ranging performance with the user's wishes.

The above embodiment can provide an image processing device that controls the display of candidate ranging positions based on evaluation values obtained from a detected image of a subject, and enables the selection of a ranging position. Note that the display of candidate ranging positions is one example, and the present invention can be applied to the display control of various information, such as a subject tracking frame, based on evaluation values for a subject area. The present invention can also be applied to the detection of multiple subjects. For example, a first and second subject may be detected. Suppose a first subject area (e.g., the entire body) is detected for the first subject, and a second subject area (part) is detected within that area. In this case, control is performed to output first information indicating that the first subject area is a second subject area to the display unit. Also, suppose a first subject area (e.g., the entire body) is detected for the second subject, and a second subject area (part) is not detected within that area. In this case, control is performed to output second information based on an evaluation value calculated for the first subject area within that area to the display unit.

(Fourth embodiment)
A fourth embodiment of the present invention will be described with reference to FIGS. 20 and 21 . In the first to third embodiments, a method of selectively controlling the display of an evaluation value using the presence or absence of detection of a ranging area of a subject was described. Specifically, when a local ranging area, such as the pupil or face of a specific subject, is detected, control is performed to disable the display of an evaluation value according to the evaluation value obtained from the image. Furthermore, when a local ranging area is not detected, control is performed to enable the display of an evaluation value according to the evaluation value obtained from the image. On the other hand, when the display of an evaluation value according to the evaluation value obtained from the image is disabled, there is a possibility that the ranging area determined solely based on the subject detection result will be an area not intended by the user. In this embodiment, a configuration is shown in which the display of an evaluation value is enabled and controlled regardless of whether a ranging area is detected by the subject detection means.

Figure 20 is a flowchart of evaluation value display control in this embodiment. In S2001, processing is performed to detect a first subject from the image on the screen. The first subject here is the entire body of a person, a car, a train, etc.

In S2002, the feature point detection unit 201 detects feature points from the area of the first subject detected in S2001. In S2003, an evaluation value is calculated according to the feature points detected in S2002. As in the previous embodiment, the evaluation value is calculated based on one or more of the saliency, density, similarity, or reliability based on similarity of the feature points in the image. The processes of S2002 and S2003 are the same as those of S302 and S303 in Figure 3.

In S2004, a process is performed to display a subject outline corresponding to the subject area detected in S2001. In S2005, the CPU 114 determines whether a second subject can be detected within the subject area detected in S2001. Here, the second subject refers to a local area such as a person's eyes or face, or the driver's seat of a train. If it is determined that the second subject can be detected, the process proceeds to S2006. If it is determined that the second subject cannot be detected, the process proceeds to S2007.

In S2006, the display unit 113 displays a subject frame corresponding to the detected second subject region. In S2007, the CPU 114 enables evaluation value display (display of ranging candidate frame) according to the evaluation value calculated in S2003. Figure 21 is a diagram showing a case in which the second subject detection result and evaluation value display are displayed side by side in this embodiment. This example shows an example in which the first subject is a train and the second subject is the train's driver's seat. A detection frame 2101 of the train, which is the first subject detection result, is displayed, and within the detection frame 2101, a detection frame 2102 of the train's driver's seat, which is the second subject detection result, and an evaluation value display frame 2103 according to the calculated evaluation value are displayed. Multiple evaluation value display frames 2103 are included in the detection frame 2101 corresponding to the first subject region, and the evaluation value calculation results are displayed in an area different from the area of the driver's seat, which is the second subject region. The evaluation value display method is not limited to a frame display method, and icon display methods, color-coded display methods, etc. may also be used.

In S2008 of FIG. 20, a process is performed to select an arbitrary area from the detection frame corresponding to the second subject area displayed in S2006 and the evaluation value display information displayed in S2007. The process of S2008 is the same as S313 of FIG. 3, and selection can be made by touch operation, button operation, stick operation, etc.

In S2009, the CPU 114 determines the ranging area based on the area selected in S2008. In S2010, the CPU 114 performs processing to display the ranging area determined in S2009 on the display unit 113. The processing in S2009 and S2010 is common to the processing in S314 and S315 in Figure 3.

In this embodiment, by controlling the evaluation value display to be enabled regardless of whether or not a ranging area is detected by the subject detection means, it is possible to determine a ranging position that reflects the user's intentions, regardless of the subject detection results. In other words, it is possible to prevent the ranging area determined based on the subject detection results from being an area that differs from the user's intention. Also, as in the previous embodiment, threshold control related to the evaluation value display allows the user to select an area with a relatively high evaluation value, making it possible to determine a ranging area with high ranging performance.

Fifth Embodiment
A fifth embodiment of the present invention will be described with reference to Figures 22 to 24. In the first to fourth embodiments, detection of a subject, detection of a ranging area candidate, a control method for displaying an evaluation value, selection of a ranging area by a user, etc. will be described. In this embodiment, in addition to processing for searching for and tracking a subject area between frames, processing for searching for and tracking a ranging area between frames when the ranging area is an area different from the subject area will be described.

Figure 22 is a flowchart illustrating the processing of subject tracking control in this embodiment. The processing of this flowchart begins when a subject is detected by the subject detection means. In S2201, the image processing unit 104 performs tracking processing of the detected subject area. The tracking processing is a process in which correlation processing is performed between the image of the subject area to be tracked and the latest frame image for frames of captured images acquired at a certain imaging cycle, and the area or position with the highest correlation is determined as the area or position of the subject moving to be tracked.

Next, in S2202, the CPU 114 determines whether a ranging area is defined other than the subject area. Regarding the ranging area, a subject area of a size suitable for ranging, which includes a part of the detected main subject area or a part of the detected main subject area, may be defined as the ranging area. Alternatively, the user may select a ranging area from candidate ranging areas determined from the evaluation value and define it as the ranging area. If a ranging area is not defined, the process ends; if a ranging area is defined, the process proceeds to S2203.

In S2203, tracking processing of the ranging area is performed. Tracking processing of the ranging area is performed by the image processing unit 104, in the same way as tracking of the subject area. In the tracking processing, correlation processing is performed between the image of the ranging area to be tracked and the latest frame image, and the area with the highest correlation is determined as the tracking result of the ranging area. A specific description will be given with reference to Figure 23.

Figure 23 shows an example of displaying the tracking results of the ranging area as a ranging frame, and an example of the range in which correlation processing is performed on the latest frame image when tracking the ranging area. Captured image 2301 is the entire captured image and is an example of a displayed image. Subject frame 2302 is an example of the main subject area detected from captured image 2301 displayed as a subject frame.

Focus frame 2303 is an example of a focus frame determined to include part of the subject area. Focus frame 2303 is set so that the user can clearly distinguish it visually from subject frame 2302, for example by changing the frame color or by using a different line style from subject frame 2302, such as a dotted line. The area within focus frame 2303 (focus frame) may be determined as the focus frame detected from captured image 2301, or may be determined by the user selecting from candidate focus frames determined from evaluation values.

The area displayed by the subject frame 2302 (subject area) and the ranging area displayed by the ranging frame 2303 can be displayed to the user in a stable manner between frames by tracking the subject area and ranging area as described in Figure 22.

The area 2304 shown in Figure 23 is an example of a search area for which correlation processing is performed on the captured image 2301 when tracking the ranging area. In the tracking processing of the ranging area in S2203 shown in Figure 22, correlation processing may be performed between the image of the ranging area and the captured image 2301, which is the latest frame image. In this case, there is a method of performing correlation processing on the search area 2304 determined based on the subject outline 2302, without performing correlation processing with the entire captured image 2301. The search area 2304 may be determined, for example, as an area of a fixed size outside the subject outline 2302, or as an area of a size that is a fixed ratio to the size of the subject outline 2302. Not using the entire frame image as the search area 2304 in tracking processing of the ranging area not only shortens processing time and reduces power consumption, but limiting the range of correlation processing improves the accuracy of tracking processing of the ranging area. FIG. 23 shows a format (frame representation format) in which the subject area is represented by a subject frame 2302 and the ranging area is represented by a ranging frame 2303, but each area may also be presented to the user in a different representation format, such as a representation format that shows only the corners of each area.

Figure 24 is a flowchart explaining the display control of the ranging area. The processing of this flowchart begins when a subject is detected by the subject detection means and subject tracking begins. In S2401, the CPU 114 determines whether a ranging area has been defined other than the subject area. As with S2202 in Figure 22, a subject area of a size suitable for ranging that includes a part of the detected main subject area may be defined as the ranging area, separate from the detected main subject area. Alternatively, the user may select from candidate ranging areas determined from the evaluation value and define it as the ranging area. If a ranging area has not been defined, the processing proceeds to S2405; if a ranging area has been defined, the processing proceeds to S2402.

In S2402, the ranging frame corresponding to the ranging area is displayed in a blinking manner for a fixed period of time. After that, the ranging frame continues to be displayed. The blinking display notifies the user that the ranging frame has started to be displayed in addition to the subject frame. The blinking display for a fixed period of time is one example, and the start of display can also be expressed in other ways, such as displaying the ranging frame in a color different from the color it is normally displayed in for a fixed period of time, or gradually darkening the display. Next, proceed to processing in S2403.

In S2403, the CPU 114 determines whether the ranging area is still being defined. If the ranging area is still being defined, the determination process in S2403 is repeated until no ranging area is defined. When the definition of the ranging area separate from the subject area has finished, the process proceeds to S2404. The definition of the ranging area ends, for example, when no subject area of a size suitable for ranging has been detected separate from the detected main subject area, or when no subject area containing part of the main subject area is detected. Alternatively, this may be the case when the user cancels the specification of the ranging area, or when the ranging area tracking control is lost, i.e., the target is lost.

In S2404, the ranging frame is flashed for a fixed period of time. After that, the ranging frame is turned off and display is terminated. The flashing display notifies the user that the ranging frame display will soon be terminated. Flashing for a fixed period of time is one example, and the end of display can be indicated in other ways, such as displaying the ranging frame in a color different from the color it is normally displayed in for a fixed period of time, or gradually fading the display. In the next step, S2405, the CPU 114 determines whether the subject area is being tracked. If the subject area is being tracked, the process proceeds to S2401, and if subject area tracking control has ended, the process of this flowchart ends.

In this embodiment, when the ranging area is defined separately from the subject area, tracking control of the ranging area is performed. Stable tracking control of the ranging area can be performed between frame images not only when the ranging area coincides with the subject area, but also when the ranging area is a separate area from the subject area. As a result, more stable ranging control can be achieved.

While the preferred embodiments of the present invention have been described, the present invention is not limited to these embodiments and various modifications, variations, and combinations are possible within the scope of the invention.

(Other embodiments)
The present invention can also be realized by supplying a program that realizes one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., an ASIC) that realizes one or more of the functions.

100: Image processing device 104: Image processing unit 112: Display control unit 113: Display unit 114: CPU

Claims (26)

a detection means for detecting a first region in the image corresponding to a subject and a second region in the image corresponding to a portion of the subject ;
a calculation means for calculating an evaluation value for each of a plurality of feature points in an image from information of the feature points;
a control means for controlling , based on the evaluation value, outputting information about a ranging area corresponding to the ranging position in the image to a display means that displays the image;
The control means controls the display means to display information indicating the first region, and to display a plurality of pieces of information indicating regions corresponding to a plurality of the evaluation values as candidates for the ranging region corresponding to the second region in a display region in the image corresponding to the first region, and controls to change the number of the plurality of pieces of information by controlling a threshold value of the evaluation value.
1. An image processing device comprising: The image processing device according to claim 1 , characterized in that the control means selectively controls the display means to output, together with information indicating the first area detected by the detection means, either first information indicating the second area detected by the detection means or second information based on the evaluation value calculated by the calculation means. The image processing device according to claim 2, characterized in that, when the first area is detected by the detection means and the second area is detected by the detection means in the first area, the control means controls the output of the first information in the area to the display means, and when the first area is detected by the detection means and the second area is not detected by the detection means in the first area, the control means controls the output of the second information in the area to the display means. 4. The image processing device according to claim 1, wherein the detection means detects the entire subject as the first region and detects a part of the subject as the second region . acquiring information about feature points in the first region ;
4. The image processing apparatus according to claim 2, wherein the control means determines an evaluation value calculated from the feature points by the calculation means and controls output of the second information to the display means. 4. The image processing device according to claim 3, further comprising: the display means for displaying the second information in a display area corresponding to the first area when the detection means does not detect the second area . 7. The image processing device according to claim 1, wherein the calculation means calculates the evaluation value based on at least one of the saliency, density, and similarity of feature points in the image, and a reliability based on the similarity. 4. The image processing device according to claim 2, wherein the control means controls the display means to output, as the first information, information on a display frame indicating that the area is the second area , or to output, as the second information, information on a display frame corresponding to the evaluation value, to the display means. The image processing device according to claim 8 , wherein the control means performs control to change the number of display frames corresponding to the evaluation value by comparing the evaluation value with a threshold value. 10. The image processing device according to claim 8, wherein the control means performs control to select one of the display frames corresponding to each of the plurality of evaluation values, and determines the distance measurement area corresponding to the selected display frame. the control means controls the display means to output, as the second information, information on a display color corresponding to the evaluation value;
4. The image processing device according to claim 2, wherein the display means displays a color distribution based on the evaluation value. The image processing device according to claim 11, wherein the control means performs control to change the display color by comparing the evaluation value with a threshold value. 13. The image processing device according to claim 11, wherein the control means performs control to select one of the display colors corresponding to each of a plurality of the evaluation values, and determines a distance measurement area corresponding to the selected display color. 4. The image processing device according to claim 2, wherein the control means determines whether to control output of the second information to the display means by comparing the size of the second area with a threshold value. 15. The image processing device according to claim 14, wherein the control means performs control to output, to the display means, information on a display frame or a display color corresponding to the evaluation value as the second information when the size of the second area is equal to or larger than a threshold value. 16. The image processing device according to claim 1, wherein the calculation means calculates a similarity from feature amounts between feature points in the image, determines a reliability based on the similarity, and determines the evaluation value based on a result of the reliability determination. 8. The image processing device according to claim 7 , wherein the control means performs control to change the number of display frames or display colors corresponding to the evaluation value by changing a threshold value for the density of feature points in the image. 18. The image processing device according to claim 17, wherein the control means controls the display means to display the display frame for the ranging area corresponding to the feature point whose saliency is higher than the threshold in the area where the density is equal to or higher than the threshold. the display means acquires the second information and displays a color distribution based on the evaluation value;
4. The image processing device according to claim 2, wherein the control means controls the display means to display a distance measurement area including a plurality of display colors corresponding to a plurality of different evaluation values. The image processing device according to claim 8 , wherein the control means controls the display means to display the display frame for a distance measurement area including a plurality of the distance measurement positions . The image processing device according to claim 4 , wherein the detection means detects the pupil, face, or head of the subject as the second region . 22. The image processing device according to claim 1, wherein the control means performs tracking control of a subject area corresponding to a detected subject by correlation processing between a first image specified based on the detected subject and an input second image. 23. The image processing device according to claim 22, wherein the control means determines a part of the detected first area as a distance measurement area and performs tracking control of the distance measurement area. An imaging device comprising the image processing device according to any one of claims 1 to 23. a detecting step of detecting a first region in the image corresponding to an object and a second region in the image corresponding to a portion of the object ;
a calculation step of calculating an evaluation value for each of a plurality of feature points in an image from information of the feature points;
a control step of controlling , based on the evaluation value, to output information about a ranging area corresponding to a ranging position in the image to a display means that displays the image;
The control method for an image processing device, characterized in that the control step includes controlling the display means to display information indicating the first area, and to display, within a display area in the image corresponding to the first area, a plurality of pieces of information indicating areas corresponding to a plurality of the evaluation values as candidates for the ranging area corresponding to the second area, and controlling a threshold value for the evaluation value to change the number of the plurality of pieces of information . A program causing a computer of an image processing apparatus to execute each step according to claim 25.