JP7163025B2 - Image measuring device, image measuring method, imaging device, program - Google Patents

Description

The present invention relates to a processing technique for calculating the length of a subject in an image measurement device.

A stereo camera that captures images for stereoscopic viewing can simultaneously capture images of the same subject using two imaging units, and acquire two types of images having parallax. There is also a method of calculating the length of an image in the depth direction based on data of two types of images captured by a stereo camera or the like. Japanese Unexamined Patent Application Publication No. 2002-100000 discloses a technique of measuring the distance between two points (the length in a three-dimensional space) on a subject image specified by a user using a stereoscopic imaging device.

On the other hand, there is known a pupil division imaging method in which a single imaging unit divides light beams from a subject that pass through different pupil partial regions of an imaging optical system to generate a plurality of pieces of image data having parallax.

Japanese Unexamined Patent Application Publication No. 2011-232330

When calculating the length in the depth direction of an image captured by the split-pupil imaging method, there is a possibility that the accuracy in calculating the length in the depth direction will decrease in regions where the defocus amount is greater than or equal to a predetermined value. The reason for this is that a large defocus amount blurs the target image and impairs the symmetry of image signals between a plurality of images having parallax. Therefore, in order to more accurately measure the length in the three-dimensional space between two points specified on the subject image, it is necessary to change the focus position at the same angle of view with respect to the subject to be measured. Multiple images taken by For example, when a plurality of captured images are used as images for distance measurement, the depth direction length of each image for distance measurement can be used for each of the two designated points for more accurate distance measurement. It is necessary to determine whether the value can be obtained.
SUMMARY OF THE INVENTION It is an object of the present invention to calculate a distance measurement value with higher accuracy in an image measuring apparatus that acquires image data and performs measurement processing.

An apparatus according to an embodiment of the present invention comprises first acquisition means for acquiring a plurality of image data captured at a plurality of focus positions and from different viewpoints, and image data pairs from different viewpoints at each focus position. second acquiring means for acquiring an image shift amount or defocus amount of a subject using a second acquiring means; designating means for specifying a plurality of distance measuring points in the image data; and the image deviation amount acquired by the second acquiring means . Alternatively, first distance information corresponding to each distance measuring point designated by the designation means is calculated from the defocus amount, and the first distance information is used to obtain the designated distance in a three-dimensional space including the depth direction of the image. calculating means for calculating the distance between the distance measuring points; and reliability determination means for determining reliability corresponding to each distance measuring point specified by the specifying means in the plurality of image data. The calculation means calculates the length between the distance measurement points from the first distance information of the distance measurement points in the selected image data using the reliability determined by the reliability determination means.

According to the present invention, an image measuring apparatus that acquires image data and performs measurement processing can calculate a distance measurement value with higher accuracy.

1 is a block diagram showing the overall configuration of an image measuring device according to this embodiment; FIG. It is a figure explaining a plurality of images in this embodiment. FIG. 4 is an explanatory diagram of an effective ranging range for each image; 10 is a flowchart for explaining the processing of the entire image measurement; 4 is a flow chart for explaining a process of selecting a focus detection point; FIG. 10 is an explanatory diagram showing an example of selection of distance measuring points; 4 is a flowchart for explaining a method of determining an evaluation value; 4 is a flowchart for explaining a method of searching for corresponding points; It is a schematic diagram explaining the search method of a corresponding point. 4 is a flowchart for explaining a method of calculating a defocus amount; It is a figure which illustrates the pixel structure of an imaging part. FIG. 5 is a schematic diagram for explaining image division processing and image shift amount calculation processing; FIG. 3 is a schematic diagram showing the relationship among defocus amount, F-number, and image shift amount; 4 is a flowchart for explaining evaluation value determination processing; It is a figure explaining various evaluation values with a graph. FIG. 3 is a schematic diagram showing an example of an effective range-finding range and a plurality of range-finding points for each image; It is a figure explaining the display processing of a distance measurement value. It is a block diagram which shows the structure of the apparatus which concerns on 2nd Embodiment. 9 is a flowchart for explaining operations in the second embodiment; 9 is a flowchart for explaining provisional distance measurement in the second embodiment; 9 is a flowchart for explaining an imaging method according to the second embodiment; It is a figure explaining the imaging|photography range which concerns on 2nd Embodiment. It is a figure explaining the evaluation value of reliability which concerns on 2nd Embodiment. FIG. 11 is a diagram illustrating an effective ranging range according to the second embodiment; FIG. It is a figure explaining the imaging|photography method which concerns on 2nd Embodiment. FIG. 10 is an explanatory diagram of a photographing method when the number of divisions of the effective distance measurement range is increased;

Preferred embodiments of the present invention will now be described in detail with reference to the drawings. The image measurement apparatus of each embodiment can be widely applied to imaging apparatuses, information processing apparatuses, and the like that acquire image data and perform measurement processing.

[First embodiment]
FIG. 1 is a block diagram showing a configuration example of an image measurement device 100 of this embodiment. The image measurement apparatus 100 includes a control unit 101 and modules (reference numerals 102 to 110), acquires image information and various information related to the image, and executes processing.

The control unit 101 includes a CPU (Central Processing Unit) and the like, and controls transmission and reception of information between modules that configure the image measuring apparatus 100 . The image input unit 102 stores data of a plurality of images input to the image measurement apparatus 100 in the memory 103 according to control commands from the control unit 101 . The memory 103 is controlled by the control unit 101 to read out information necessary for processing in each module and store processing results by each module.

A shooting information acquisition unit 104 acquires shooting information and outputs it to the memory 103 according to a control command from the control unit 101 . Information storage in the memory 103 is performed via the control unit 101 . The imaging information is information associated with a plurality of images input to the image measurement device 100 and generated at the time of imaging.

A focus detection point selection unit 105 acquires focus detection point information input to the image measuring apparatus 100 and selects two focus detection points for data of a plurality of images stored in the memory 103 . A focus detection point selection unit 105 outputs each coordinate information of the selected two points to the memory 103 according to a control command from the control unit 101 . Each piece of coordinate information is stored in memory 103 via control unit 101 .

The corresponding point searching unit 106 reads data of a plurality of images stored in the memory 103 . A process of searching for each ranging point corresponding to the ranging points (two points) selected by the ranging point selection unit 105 is performed on the read multiple image data. The corresponding point searching unit 106 calculates coordinate information corresponding to each distance measuring point in the plurality of images. The calculated coordinate information is stored in the memory 103 via the control unit 101 .

A defocus amount calculation unit 107 reads data of a plurality of images and shooting information stored in the memory 103 . A defocus amount is calculated at coordinates corresponding to each of the distance measurement points (two points) for data of a plurality of images. The calculated defocus amount is stored in the memory 103 via the control unit 101 .

The evaluation value determining unit 108 reads the data of the plurality of images, the defocus amount, and the coordinate information corresponding to the distance measurement points (two points) in the plurality of images, which are stored in the memory 103 . An evaluation value of coordinates corresponding to each ranging point in a plurality of images is determined. The determined evaluation value is stored in memory 103 via control unit 101 .

The inter-focus point distance calculation unit 109 reads out the evaluation value and the defocus amount stored in the memory 103 . An image to be used is determined from among the plurality of images based on the read evaluation value. Based on the defocus amounts at the determined focus points (two points) in the image, processing is performed to calculate the coordinates of each focus point in the three-dimensional space. An inter-focus point distance calculator (hereinafter simply referred to as a distance calculator) 109 calculates the distance between the focus points from the coordinates of each focus point in the three-dimensional space. The calculated length information is stored in memory 103 via control unit 101 .

An image measurement result display unit (hereinafter simply referred to as a display unit) 110 displays an image based on the data of a plurality of images read from the memory 103, the coordinate information of the distance measurement points, and the length information between the distance measurement points. The measurement results are displayed on the screen and presented to the user. For a representative image selected from a plurality of images, the display unit 110 displays the coordinates of two points that are distance measurement points and the distance between the two points.

A plurality of images input to the image measurement device 100 will be described in detail with reference to FIG. FIG. 2A is a schematic diagram illustrating the positional relationship between the imaging device and the subject. The imaging device 201 includes a split-pupil imaging unit. For example, it is possible to generate first and second viewpoint images as a plurality of images having parallax (parallax images) by splitting a light beam from an object into two and receiving the light beams passing through the pupil partial regions of the imaging optical system. can. The image capturing apparatus 201 captures an image of the subject 202 while changing the focus position corresponding to the in-focus position and the depth of field, and acquires data of a plurality of images. That is, image data pairs of a plurality of parallax images are obtained at a plurality of focus positions. In FIG. 2A, regarding the imaging device 201, the horizontal direction is defined as the x-axis direction, the vertical direction is defined as the y-axis direction, and the depth direction is defined as the z-axis direction. The depth direction is the direction along the optical axis direction of the imaging device 201 .

FIG. 2B is an explanatory diagram of a plurality of images captured by the imaging device 201. FIG. As shown in FIG. 2A, photography is performed with the positional relationship between the imaging device 201 and the subject 202, and the multiple images that have been photographed include an image for range-finding point selection and an image for range-finding. . The focus point selection image is an image obtained by shooting with a deep depth of field in order to select a focus point. Also, the image for distance measurement is an image of the same subject photographed at different focal positions. The range-finding point selection image is an image having a wider focus area than the range-finding image. By selecting a specific image from among the range finding images, the range finding point selection image can also be used as the range finding image. Furthermore, if there is no image with a wider focus area than the image for distance measurement, it is also possible to arbitrarily select one image for distance measurement point selection from the images for distance measurement.

FIG. 3 is a diagram for explaining the effective ranging range for each ranging image. In the positional relationship between the imaging device 201 and the object 202 shown in FIG. show. The axis perpendicular to the plane of FIG. 3 is defined as the y-axis, and the two orthogonal axes within the plane of the paper are defined as the x-axis and the z-axis. The focus position can be expressed as a focus plane with respect to the z-axis by looking down from the y-axis direction. For the images for distance measurement that are shot multiple times with different focus positions, the focus planes in each image for distance measurement are set to focus plane 203p, focus plane 204p, focus plane 205p, and focus plane 206p in order from the one closest to the camera position. . In this case, the effective distance measurement range on the front side and the effective distance measurement range on the back side are shown in each focal plane. For example, in the focus plane 203p, the effective distance measurement range on the near side of the focus plane 203p is the range from the focus plane 203p to a position 203b indicating the effective distance measurement range. Further, the effective distance measurement range on the far side of the focus plane 203p is a range from the focus plane 203p to a position 203i indicating the effective distance measurement range. The range from the position 203b indicating the effective range-finding range on the front side to the position 203i indicating the effective range-finding range on the back side is the effective range-finding range 203a on the focus plane 203p. Similarly, the effective ranging ranges of focus planes 204p, 205p, and 206p are ranges 204a, 205a, and 206a, respectively. The effective distance measurement range is a range in which the effectiveness of distance measurement accuracy in the depth direction centering on the focus position is high, and can be determined based on the defocus amount described later. In other words, the ranging accuracy outside the effective ranging range is much lower than the ranging accuracy within the effective ranging range.

The processing performed by the image measurement device 100 will be described in detail with reference to the flowchart shown in FIG. In S<b>301 , information on a plurality of images, shooting information, and ranging point information are acquired and input to the image measurement apparatus 100 . The plurality of images include an image for range finding point selection and an image for range finding, and the image input unit 102 acquires them. As the shooting information, at least the following information (a) to (h) is acquired by the shooting information acquisition unit 104 for each of a plurality of images.
(a) the distance from the imaging device to the focus position (b) the pixel pitch of the imaging device (c) the conversion coefficient determined by the size of the opening angle of the center of gravity of the light flux passing through the pair of ranging pupil regions (d) the F value ( e) pupil distance (f) distance from the image principal point of the lens of the imaging device to the imaging sensor surface (g) distance from the object principal point of the lens of the imaging device to the subject (h) lens state information of the imaging device (focal length and lateral magnification ratio)
The shooting information is necessary when calculating the defocus amount and when calculating the coordinates in the three-dimensional space.

The focus detection point information is information for determining whether the focus detection point is automatically selected or arbitrarily selected by the user, and is acquired by the focus detection point selection unit 105 . Further, the distance measuring point information includes information on the determination method when the method of selecting the distance measuring point is automatically determined.

In the next step S302, a process of selecting focus detection points (two points) is executed. Details of the processing will be described later with reference to FIG. In S303, a process of determining the evaluation value of each distance measuring point is executed. Details of the processing will be described later with reference to FIG. In S304, processing is executed to calculate the three-dimensional coordinates of each ranging point and the length between the two points based on the evaluation values determined in S303, and the calculation results are presented to the user. The evaluation method will be described later with reference to FIG. 16 . After S304, the process ends.

Focus detection point selection processing by the focus detection point selection unit 105 will be described with reference to FIG. FIG. 5 is a flow chart for explaining the process of selecting a focus detection point. In S401, a process of reading out image data for selecting a focus point is performed. In the next step S402, the focus detection point information is referenced, and determination processing is executed to determine whether or not the focus detection point is designated by the user. If the focus detection point is not specified by the user's operation, the process proceeds to S403. Also, if the focus detection point is specified by the user's operation, the process proceeds to S404.

In S403, the focus detection point is automatically determined based on the conditions set in advance by the focus detection point information. A specific example will be described with reference to FIG. FIG. 6A is a schematic diagram for explaining feature point detection and evaluation value calculation. First, feature point detection is performed on an image 401 for selecting a distance measuring point, and feature points 4011 to 4017 of the subject having a rectangular parallelepiped shape, for example, are obtained. At this time, the evaluation value at the point detected as the feature point is calculated, and the feature amount of the feature point is evaluated. In FIG. 6A, the larger the feature amount of the feature point, the larger the feature point (black point) is displayed. That is, it is assumed that the larger the feature amount, the larger the radius of the black dot mark representing the feature point displayed in the range-finding point selection image 401 . For example, among the feature points 4011-4017, the feature points 4011 and 4012 have higher feature amounts than the other feature points 4013-4017. In this case, feature points 4011 and 4012 are determined as two distance measurement points. Methods for detecting feature points include a corner detection method, a contrast detection method, an edge detection method, and the like. There are a method of using these methods independently and a method of calculating feature points by combining a plurality of methods.

In S404 of FIG. 5, two points specified by the user are selected as focus detection points based on conditions set in advance by focus detection point information. A specific example will be described with reference to FIG. FIG. 6B is a schematic diagram for explaining a selection operation by the user. An example in which the user designates focus detection points 4021 and 4022 for the focus detection point selection image 401 is shown. The user can perform an operation of selecting a desired range-finding point on the touch panel with his or her finger or an instruction operation with the pointing device and cursor graphic 4023 . As a result, two distance measuring points are selected. In addition to the example shown in FIG. 6B, it is also possible for the user to select coordinates on the image and set a point corresponding to the selected coordinates as the range-finding point.
As described above, the user can freely select the range-finding point, and in the case of automatic selection of the range-finding point, the user's trouble can be reduced.

Next, referring to the flowchart of FIG. 7, the process of determining the evaluation value of each distance measuring point (FIG. 4: S303) will be described. In S501, a corresponding point search is performed to search for a corresponding point in the image for distance measurement with respect to the distance measurement point selected in the image for distance measurement point selection. In S502, a process of calculating the defocus amount at the searched corresponding point is performed. In S503, an evaluation value at the distance measuring point is determined based on the calculated defocus amount and reliability. Details of the processing shown in each step of FIG. 7 will be described below.

First, in S501, since it is necessary to specify the coordinates of the distance measurement points (two points) selected in the image for distance measurement point selection in a plurality of images for distance measurement, search processing for corresponding points is performed. The processing performed by the corresponding point searching unit 106 will be described in detail with reference to the flowchart of FIG.

In S601, processing for selecting pixels around each focus point in the image for focus point selection is performed. In S602, block matching processing is performed on each ranging image using the selected peripheral pixels. A specific example will be described with reference to FIG.

FIG. 9A is a schematic diagram for explaining the search for corresponding points, exemplifying a range-finding point selection image 601 and a range-finding image 602 . In the focus detection point selection image 601 , processing is first performed to select pixels around the focus detection point 6011 as a correlation window 6012 . Correlation window 6012 is shown in an enlarged view. Furthermore, processing for selecting the correlation window 6022 selected in the window range 6021 is performed on the ranging image 602 . A selected correlation window 6022 is shown in an enlarged view. Block matching processing is performed with these correlation windows 6012 and 6022 . That is, a process of calculating the amount of error between correlation windows is executed, and this process is repeated over the entire image while changing the position of the window range 6021 . Then, from the coordinates of the range in which the calculated error amount is the smallest, it is possible to calculate the ranging point in the ranging image 602 that corresponds to the ranging point selected in the ranging point selection image 601 . As for the ranging point 6013, similarly to the ranging point 6011, a ranging point corresponding to the ranging point selected in the ranging point selection image is calculated in each ranging image.

FIG. 9B is an explanatory diagram of a method for improving the block matching process. As a method of devising the block matching processing described with reference to FIG. 9A, there is a method of narrowing down the search range in the image to reduce the amount of calculation. That is, in FIG. 9A, the entire range finding image 602 is set as the search range, but in FIG. Utilize the fact that it is a photographed image. It is unlikely that the coordinates of the ranging point in the ranging point selection image will deviate greatly from the coordinates of the ranging point in the ranging image 603 . Using this characteristic, the start position of the window range 6031 in the image for distance measurement 603 is set as the coordinates of the distance measurement point in the image for distance measurement point selection, and processing for selecting the correlation window 6032 is performed. Furthermore, the range of changing the position of the window range 6031 is restricted within the window movement range 6033 . As a result, wasteful processing due to searching for regions with low correlation can be eliminated, and the amount of computation can be significantly reduced.

The defocus amount calculation process shown in S502 of FIG. 7 will be described in detail with reference to FIG. FIG. 10 is a flowchart for explaining the processing performed by the defocus amount calculation unit 107. FIG. In S701, processing is performed to generate a pair of image data from one ranging image. Details thereof will be specifically described with reference to FIG. 11 .

FIG. 11A is a schematic diagram for explaining the pixel arrangement of an imaging device used when capturing parallax images. The axis perpendicular to the plane of the paper is the Z-axis, and the two orthogonal axes in the plane of the paper are the X-axis and the Y-axis, respectively. The imaging device that constitutes the imaging unit has a structure in which a large number of pixel units 712 are regularly arranged in a two-dimensional array. FIG. 11B is an enlarged view of one of the pixel portions 712. FIG. The pixel unit 712 includes a microlens 711 and a pair of photoelectric conversion units 713A and 714B (hereinafter referred to as pupil division pixels 713A and 714B). A plurality of photoelectric conversion units corresponding to the respective microlenses receive light beams from the subject that pass through different pupil partial regions of the imaging optical system, perform photoelectric conversion, and output electrical signals. In this embodiment, an image obtained by the pupil-dividing pixels 713A is called an A image, and an image obtained by the pupil-dividing pixels 714B is called a B image. An image obtained by synthesizing the A image and the B image is called an AB image. The AB image corresponds to an image captured by a non-pupil-splitting imaging device.

In the configuration illustrated in FIG. 11, the data of the A image and the B image can be obtained as information necessary for performing image measurement processing. Although the number of divisions of the photoelectric conversion unit is 2 in this example, the number of divisions may be 3 or more, and a plurality of viewpoint images having different viewpoints can be acquired.

In S702 of FIG. 10, a process of dividing the pair of image data generated in S701 into minute blocks is performed. A specific example will be described with reference to FIG. FIG. 12A is a schematic diagram showing data of an A image 721 and a B image 722 generated from signals respectively acquired by pupil division pixels 713A and 714B. The A image 721 and the B image 722 are divided into a plurality of blocks as indicated by minute blocks 723 . For example, it is assumed that the pixel of interest is set at the center position of the minute block for each pixel of the input image. A minute block 723 in FIG. 12A is divided so as to include a pixel of interest at a distance measuring point 724 in the center and eight pixels surrounding it.

FIG. 12B is a schematic diagram showing an A image 721 for explaining the shape of the minute blocks. A minute block 725 shown in the left diagram is an example larger in size than the minute block 723 in FIG. 12(A). Also, the minute block 726 shown in the right figure is an example of an L shape. In this way, there are no restrictions on the size and shape of the minute blocks to be divided. In addition, the regions of adjacent minute blocks may overlap each other.

In S703 of FIG. 10, correlation calculation processing is performed on the minute blocks divided in S702, and the amount of image shift at each distance measuring point is calculated. Specifically, a pair of pixel data in a minute block are denoted by E(1) to E(m) and F(1) to F(m), respectively. m represents the number of data. In this case, while relatively shifting the second data series F(1) to F(m) with respect to the first data series E(1) to E(m), two A correlation amount C(k) in a shift amount (denoted as k) between data strings is calculated.
C(k)=Σ|E(n)−F(n+k)| (1)
In equation (1) || is the absolute value symbol and the Σ operation (accumulation) is computed over n. In this Σ calculation, the range of n and n+k is limited to the range of 1 to m. The value of the shift amount k is an integer and is a relative shift amount in units of detection pitches of a pair of image data.

An example of the calculation result of Expression (1) will be described with reference to the graph shown in FIG. The horizontal axis of FIG. 12C represents the amount of image shift, and the vertical axis represents the amount of correlation C(k). The correlation amount C(k) is minimized at the image shift amount with high correlation between the pair of data sequences. Then, the image shift amount (denoted as x) that gives the minimum value C(x) for the continuous correlation amount is calculated using the three-point interpolation method according to the following formulas (2) to (5). The value of the image shift amount x is a real number.
x=kj+D/SLOP (2)
C(x)=C(kj)−|D| (3)
D={C(kj−1)−C(kj+1)}/2 (4)
SLOP=MAX{C(kj+1)-C(kj), C(kj-1)-C(kj)} (5)
Here, kj is k when the discrete correlation amount C(k) is minimized.
The amount x obtained from the formula (2) is set as the image shift amount in the pair of split-pupil images. Note that the unit of the image shift amount x is pixels.

In S704 of FIG. 10, the defocus amount (denoted as DEF) of the subject image plane with respect to the planned imaging plane is calculated by the following formula (6) from the image shift amount x at the distance measuring point obtained by the formula (2). be done.
DEF = KX · PY · x (6)
In Expression (6), PY is the pixel pitch of the image sensor (the inter-pixel distance between pixels forming the image sensor). KX is a conversion coefficient determined by the size of the opening angle of the center of gravity of the luminous flux passing through the pair of distance measuring pupil partial areas, and the unit is mm/pixel. It should be noted that the size of the opening angle of the center of gravity changes according to the size (F value) of the diaphragm opening of the lens, and is determined according to the lens information. A specific description will be given with reference to FIG.

FIG. 13 is a schematic diagram showing the relationship between the defocus amount, the F-number, and the image shift amount with respect to the intended imaging plane. From the left side of FIG. 13, an object plane 741 to be imaged, a lens 742, an expected imaging plane 743 of the imaging element, and an image plane 744 are shown. An image plane 744 is a plane located at a position corresponding to a defocus amount 745 from an expected imaging plane 743 . Image shift amounts 746 and 747 corresponding to F-numbers are shown, respectively. The image shift amount 746 is the image shift amount when the F number is small (open side), and the image shift amount 747 is the image shift amount when the F number is large (diaphragm side). It can be seen that the amount of image shift with respect to the image plane in the defocused state increases on the wide open side and decreases on the aperture side. In other words, even if the defocus amount is the same, the image shift amount differs depending on the F number. Therefore, when calculating the defocus amount from the image shift amount, it is necessary to multiply by a conversion coefficient KX, which is a gain corresponding to the F number, as shown in Equation (6).

As described above, the defocus amount can be calculated by repeating the calculation while shifting the position of the pixel of interest by one pixel. By applying this calculation processing method to the distance measurement points (two points) of each distance measurement image, the defocus amount of each distance measurement image can be calculated.

The evaluation value determination process shown in S503 of FIG. 7 will be described with reference to FIG. FIG. 14 is a flowchart for explaining the processing performed by the evaluation value determination unit 108. FIG. S801 is judgment processing regarding the first condition. The first condition is that there is an image for distance measurement (hereinafter referred to as a condition-conforming image) in which two points, which are distance measurement points, are within the effective distance measurement range of one image. If the first condition is satisfied, the process proceeds to S802, and if the first condition is not satisfied, the process proceeds to S809. Here, the effective ranging range will be described in detail. Whether or not there is a condition-satisfactory image can be determined by evaluating the defocus amount DEF calculated by Equation (6). That is, as the defocus amount DEF increases, the distance measurement accuracy of DEF decreases. A range of the defocus amount DEF in which a certain level of distance measurement accuracy is guaranteed is defined as |DEF|<Cr. Cr is a defocus amount threshold. If the defocus amount DEF that satisfies the inequality |DEF|<Cr is calculated at a range-finding point, it is determined that this range-finding point is within the effective range-finding range. That is, it is possible to determine whether or not there is a condition-conforming image depending on whether or not the magnitude of the defocus amount DEF at the distance measuring point is smaller than the predetermined threshold value Cr. Based on the defocus amount calculated for each ranging point of a plurality of ranging images, it is possible to evaluate whether or not a condition-conforming image exists.

As described above, the defocus amount changes according to the aperture size (F value). Referring to FIG. 13, specifically, the first ranging image with a large F-number has a wider effective range-finding range than the second ranging image with a small F-number. can be done. However, the accuracy of distance measurement is lowered in the first image for distance measurement. Using this characteristic, when there is an image for ranging with a large F-number, provisional ranging is performed for the ranging points (2 points) with this one ranging image, and the two ranging points are It can be determined whether or not it is within the effective ranging range of one image. By doing so, it is possible to reduce the amount of calculation and the time required compared to the process of evaluating whether or not the two ranging points in each ranging image are within the effective ranging range.

In S802 of FIG. 14, it is determined whether or not the reliability of accuracy is prioritized as an evaluation value. Accuracy represents the degree of correspondence to the measurement error from the true value, and the closer the measured value is to the true value, the higher the reliability. Confidence in accuracy is determined by evaluating the amount of focus shift. The focus movement amount indicates the focus movement amount between the distance measurement images when each distance measurement image for calculating the depth information at the two distance measurement points is selected. When the amount of focus movement is large, the measurement error of the distance to the focus position when photographing a plurality of images becomes large. Hypothetically, in an implementation method that always prioritizes the reliability of accuracy as an evaluation value without judging whether to prioritize the reliability of accuracy as an evaluation value, regardless of the quality of the image quality, only the amount of focus movement The evaluation value is calculated with . A problem that occurs in this case is a decrease in accuracy of distance measurement. For example, an image with low contrast at the focus point or an image with saturated luminance at the focus point will be selected as the image for range finding because the amount of focus movement is small, resulting in poor range accuracy. may decline. On the other hand, in an implementation method that does not prioritize the reliability of accuracy as an evaluation value, all other evaluation values must be calculated or determined, which may increase the computational load. For the above reasons, in S802, it is determined whether or not the reliability of accuracy is given priority as an evaluation value. , the process proceeds to S806. As the determination method, for example, there are a user-based method and a device-based method as follows.

-Method of arbitrarily setting by user's operation The device performs priority determination according to the setting instruction performed by the user using the operation unit.
・Method for the device to automatically determine Automatically determine that the degree is not given priority as an evaluation value.
Each method may be performed independently, or a plurality of methods may be used according to the conditions.

In S803, the reliability of the accuracy of each distance measurement image of the distance measurement points (two points) is determined by evaluating the amount of movement of the focus. When the amount of focus movement is large, the measurement error of the distance to the focus position when photographing a plurality of images becomes large. Therefore, the evaluation value determination unit 108 reduces the evaluation value of the focus movement amount. Also, when the distance measurement images for calculating the depth information at the distance measurement points (two points) are the same distance measurement image, the focus shift amount is zero. In such a case, the evaluation value determination unit 108 increases the evaluation value of focus movement. A specific example will be described with reference to FIG.

FIG. 15A is a graph for explaining the relationship between the amount of focus movement and the corresponding evaluation value. The horizontal axis represents the amount of focus movement (denoted as pm), and the vertical axis represents the evaluation value (denoted as PIN). As the focus movement amount pm increases, the evaluation value PIN decreases. For example, the evaluation value PIN is calculated by Equation (7).
PIN = b1 - b2 x pm (7)
However, both the intercept b1 and the coefficient b2 are positive values (b1, b2>0) and can be set arbitrarily. When the focus movement amount pm is not evaluated, PIN = b3 (8) as shown in Equation (8)
and b3 is a constant value (positive value) and can be arbitrarily set.

Denote the reliability of accuracy in each ranging image as Rs,
Rs = PIN (9)
and When evaluating the focus movement amount pm, it can be determined so that the reliability Rs decreases as the value of pm increases. As a result, it is possible to reduce the measurement error due to the movement of the focus, so that it is possible to calculate a more accurate distance measurement value. It should be noted that the evaluation value PIN can also be determined by providing a reference table using parameters instead of calculating the evaluation value PIN using the formulas (7) and (8). Furthermore, it is also possible to determine the reliability of accuracy Rs from the relationship between the focus movement amount pm and the evaluation value PIN by providing a reference table with parameters instead of calculating with the formula (9). .

In S804 of FIG. 14, the reliability of the reproducibility of each ranging image for the two ranging points is determined. The reliability of reproducibility is the reliability determined by evaluating the defocus amount or image shift amount, luminance value, and variance value at each of the two distance measuring points. A specific description will be given with reference to FIGS. 15B to 15D.

FIG. 15B is a graph for explaining the reproducibility reliability based on the evaluation value of the defocus amount. The evaluation value of the defocus amount DEF shown on the horizontal axis is expressed as DFR and shown on the vertical axis. As the absolute value of the defocus amount |DEF| increases, the blurring of the image increases, so that the distance measurement accuracy decreases even within the effective distance measurement range. The defocus amount in each distance measurement image of two distance measurement points is calculated using equations (1) to (6). With the position 1501b at which the defocus amount DEF is zero as a reference, the evaluation value DFR decreases as the absolute value of the defocus amount DEF increases. The evaluation value DFR can be expressed as Equation (10).
DFR=c1−c2×|DEF| (10)
In equation (10), c1 is the DFR value at location 1501b. Let c1>0, which is the maximum value of DFR. Also, c2 is an arbitrary coefficient, and c2>0. However, the values of c1 and c2 are set so that DFR>0 regardless of the value of DEF.
As described above, it is possible to decrease the evaluation value DFR as the absolute value |DEF| of the defocus amount increases. As a result, it is possible to reduce the measurement error caused by the defocus amount, so it is possible to calculate a more accurate distance measurement value.

FIG. 15C is a diagram for explaining the reliability of reproducibility based on the evaluation value of the luminance average value. The evaluation value LMR is shown on the vertical axis with respect to the luminance average value LUM shown on the horizontal axis. As for the luminance values of the two ranging points in each ranging image, the average value of the luminance values of the pixels near the coordinates of each of the two ranging points in each ranging image is calculated. The neighborhood range can be set arbitrarily. When the average luminance value is relatively large (or small) with respect to the intermediate value of the range expressing luminance, it indicates that the average luminance value is brighter (or darker) than the intermediate value of luminance. If the average luminance value is too bright or too dark compared to the intermediate luminance value, an error may occur when searching for the distance measurement point of each distance measurement image, and the defocus amount calculation error may occur. An increase in these errors degrades ranging accuracy. In the example shown in FIG. 15C, the luminance average value LUM shown on the horizontal axis is expressed in the range of 0 to 255. In the example shown in FIG. Using the position 1501c corresponding to the intermediate value 127 as a reference, the evaluation value LMR decreases as the luminance average value LUM increases (or decreases). The evaluation value LMR can be expressed as Equation (11).
LMR = d1 - d2 x |((LM/2)-LUM)| (11)
d1 shown in equation (11) is the value at position 1501c corresponding to the intermediate value 127, where d1>0, which is the maximum value of LMR. The value of the coefficient d2 can be set arbitrarily, and d2>0. LM is the maximum representable luminance value, and FIG. 15(C) shows an example of 255 values.
As described above, it is possible to decrease the evaluation value LMR as the average luminance value LUM diverges from the median value of the luminance values that can be expressed. As a result, it is possible to reduce the measurement error caused by the luminance average value, so that the distance measurement value can be calculated with higher accuracy.

FIG. 15D is a diagram for explaining the reliability of reproducibility based on the evaluation value of the variance. The evaluation value is indicated as CNR and shown on the vertical axis with respect to the variance value CON of the image shown on the horizontal axis. The variance of the two ranging points in each ranging image can be calculated from the variance of pixels in the vicinity of the coordinates of each of the two ranging points in each ranging image. The neighborhood range can be arbitrarily set at the time of calculation. When the variance value is small, that is, when the contrast is low, an error occurs when searching for the distance measurement point of each distance measurement image, and an error occurs in calculating the defocus amount, resulting in a decrease in distance measurement accuracy. descend. As shown in FIG. 15(D), the evaluation value CNR increases as the variance value CON increases. CNR can be expressed by Equation (12).
CNR=e1×CON (12)
In equation (12), e1 is a proportionality factor for controlling the value of CNR. Since the variance CON satisfies CON≧0, the CNR is expressed as a value greater than or equal to zero.
As described above, it is possible to increase the evaluation value as the variance value increases. As a result, it is possible to reduce the measurement error caused by the variance value, so that the distance measurement value can be calculated with higher accuracy.

Denoting the reliability of reproducibility as Rr, it is calculated as the product of three evaluation values as in Equation (13) according to Equations (10), (11) and (12).
Rr = DFR x LMR x CNR (13)
That is, the reproducibility reliability Rr in the reliability determination process can be determined by evaluating the defocus amount DEF, the luminance average value LUM, and the variance value CON. In the present embodiment, the method of calculating the evaluation value DFR of the defocus amount, the evaluation value LMR of the luminance average value, and the evaluation value CNR of the variance value using the formulas (10), (11), and (12) has been described. . The method is not limited to this, and a method of determining each evaluation value by preparing a reference table based on parameters in advance may be adopted. Further, the reproducibility reliability Rr is not limited to the calculation method using the formula (13), and may be determined using a parameter lookup table. In this case, the reliability Rr can be determined based on the relationship among the evaluation values DFR, LMR, and CNR.

In S805, an evaluation value for each distance measurement image of the two distance measurement points is determined from the reliability of accuracy Rs and the reliability of reproducibility Rr. This evaluation value (denoted as R) is calculated by Equation (14).
R = (s x Rs) x (r x Rr) (14)
In Equation (14), s and r are weighting parameters for reliability Rs and Rr, respectively. Arbitrary values of s≧0 and r≧0 are set.
As described above, when there is a condition-satisfactory image and the reliability of accuracy Rs is not prioritized as an evaluation value, from each reliability of accuracy and reproducibility, An evaluation value can be determined.

Next, a case in which the reliability Rs of accuracy is prioritized as an evaluation value will be described. In this case, the process proceeds from S802 to S806. In S806, determination processing is performed for the second condition. The second condition is that there are multiple condition-matching images. If the second condition is not satisfied, that is, if it is determined that there are not a plurality of condition-satisfactory images, the process advances to S807. If the second condition is satisfied, the process proceeds to S808. In S807, the evaluation value is determined so that the condition-satisfactory image has the maximum evaluation value. That is, in a plurality of ranging images, the evaluation value R of the ranging points (two points) in the ranging image where the two points as ranging points do not exist within the effective ranging range of one image is It is synonymous with setting it to zero. The evaluation value R at the range-finding point in the range-finding image that is not the condition-matching image is set as s=0 in Equation (14).
As described above, when there is one condition-satisfactory image and priority is given to the reliability of accuracy as the evaluation value, the evaluation value can be determined so that the condition-satisfactory image has the maximum evaluation value.

On the other hand, in S808, a process of excluding ranging images other than the condition-matching images from the acquired ranging images is performed. As a result, it is possible to reduce the computational load required to calculate the reliability of each ranging image.

In S809, processing is performed to determine the reliability of the reproducibility of each ranging image for the two ranging points. The reliability of reproducibility can be calculated in the same manner as the reliability of reproducibility Rr obtained by formulas (10) to (13). However, if it is determined in S801 that there is no condition-satisfactory image, the reproducibility reliability Rr is calculated for all ranging images. On the other hand, if it is determined in S806 that there are a plurality of condition-satisfactory images, the reproducibility reliability Rr is calculated only for the condition-satisfactory images.

After S809, the process advances to S810 to calculate an evaluation value R for each ranging image for each ranging point from the reproducibility reliability Rr. The evaluation value R is calculated using Equation (15).
R = r x Rr (15)
r is a coefficient parameter for the reliability Rr and is set to an arbitrary value of r≧0.
As described above, it is possible to determine the evaluation value of each range-finding image for each range-finding point when there is no condition-conforming image (when proceeding from S801 to S809). In addition, even when there are a plurality of condition-satisfactory images and priority is given to the reliability of accuracy as the evaluation value, the evaluation value of each range-finding image for each range-finding point can be determined.

An evaluation method based on the evaluation value determined in S303 of FIG. 4 will be described with reference to FIG. FIG. 16 is an explanatory diagram exemplifying specific range-finding points A, B, and C for the effective range-finding range described in FIG. A case of determining each evaluation value between the distance measuring points A and B, between the distance measuring points A and C, and between the distance measuring points B and C will be exemplified.

Table 1 shows a specific example of determining evaluation values for each of the distance measuring points A and B with respect to the positional relationship between the distance measuring points A and B. A natural number variable n (=1, 2, . . . ) represents an identification number assigned to distinguish each ranging image. The value of n corresponds to the numbers in parentheses in the effective ranging ranges (1) to (4) shown in FIG. "No focus movement R" is an evaluation value when the amount of focus movement is zero, and "With focus movement R" is an evaluation value when the amount of focus movement is not zero. Focusing point A is within effective ranging range (4), ranging point B is within effective ranging range (1), and ranging point C is within effective ranging ranges (1) and (2). Assume that there is

In the example of FIG. 16, there is no distance measurement image in which the distance measurement points A and B are within the effective distance measurement range of one image. Therefore, the evaluation value R without focus movement cannot be determined, and processing is performed to determine only the evaluation value R with focus movement. For example, the maximum evaluation value R (=80) at the range-finding point A is determined in the range-finding image shot in the effective range-finding range with n=4. Also, the maximum evaluation value R (=80) at the range-finding point B is determined in the range-finding image shot in the effective range-finding range with n=1. That is, regarding the ranging image used to calculate the length between the ranging points A and B, the evaluation value for the ranging point A is calculated from n=4 ranging images. For B, the evaluation value is calculated from the n=1 distance measurement image.
Table 2 shows a specific example in which each evaluation value of the distance measuring points A and C is determined with respect to the positional relationship between the distance measuring points A and C.

In the example of FIG. 16, there is no distance measurement image in which the distance measurement points A and C are within the effective distance measurement range of one image. Therefore, the evaluation value R without focus movement is not determined, and only the evaluation value R with focus movement is determined. The maximum evaluation value R (=80) at the ranging point A is determined from the ranging image shot in the effective ranging range with n=4. The distance measurement point C is included in the effective distance measurement range (1) corresponding to n=1 and the effective distance measurement range (2) corresponding to n=2. A certain n=2 ranging image can be determined. That is, regarding the ranging image used to calculate the length between the ranging points A and C, the evaluation value for the ranging point A is calculated from n=4 ranging images, and the ranging point As for C, an evaluation value is calculated from n=2 ranging images.
Table 3 shows a specific example in which each evaluation value of the distance measuring points B and C is determined with respect to the positional relationship between the distance measuring points B and C.

In the example of FIG. 16, there is a distance measurement image in which the distance measurement points B and C are within the effective distance measurement range of one image. Both the evaluation value R without focus shift and the evaluation value R with focus shift can be determined. Regarding the distance measuring point B, the evaluation value R at n=1 is the maximum for both R with focus movement and R without focus movement. Therefore, as the image for distance measurement to be used for the distance measurement point B, the image for distance measurement with n=1 can be determined. For the distance measuring point C, the maximum evaluation value is obtained at n=1 in the case of R without focus movement, and the maximum evaluation value is obtained at n=2 in the case of R with focus movement. In such a case, a process of comparing the product of the evaluation values in the case of R with focus movement and the product of the evaluation values in the case of R without focus movement is performed. Specifically, if the evaluation value product in the case of R with focus shift is denoted by PA, it is calculated as shown in Equation (16).
PA = 80 x 80 = 6400 (16)
On the other hand, if the evaluation value product in the case of R without focus movement is denoted by PB, it is calculated as shown in Equation (17).
PB = 90 x 80 = 7200 (17)
Therefore, since PB>PA, the image for distance measurement with n=1 is used as the image for distance measurement used for distance measurement points B and C based on the evaluation value of no focus shift R. can decide.

Next, processing for calculating the three-dimensional coordinates of each ranging point and the length between two points from the ranging image determined based on the evaluation value will be described. Distance measurement in a three-dimensional space based on the shooting information acquired by the shooting information acquisition unit 104, the distance measurement point selected by the distance measurement point selection unit 105, and the defocus amount DEF obtained from the image for distance measurement. The coordinates of the point are calculated. These coordinates are expressed as (x, y, z) and can be calculated by formulas (18) to (22). However, as a three-dimensional coordinate system, the center of the exit pupil is the center of the optical axis, and the axis along which the defocus amount DEF is zero is the z-axis.
(h, v) = (hb, hv) x PY (18)
(H, V) = (h, v) × (1-DEF/L) (19)
(x, y) = (H, V) x |dist/(dist_d + DEF) | (20)
dist=1/(1/(dist_d+DEF)-1/f) (21)
z=length−dist (22)
The meaning of each symbol is as follows.
(h, v): Coordinates on image sensor surface of focus point (hb, hv): Coordinates on image of focus point PY: Pixel pitch L: Pupil distance (H, V): Coordinates on image sensor surface Coordinates on the xy plane defocused from (h, v) dist: Distance from the object principal point on the lens of the imaging device to the subject dist_d: Distance from the image principal point on the lens of the imaging device to the imaging sensor surface f: Imaging optics focal length of the system length: the distance from the imaging sensor surface of the imaging device to the subject (subject distance)

The distance calculation unit 109 calculates the three-dimensional coordinates (x, y, z) of the distance measurement points (two points) using formulas (18) to (22). However, when the distance measurement images for calculating the coordinates of the distance measurement point are different for each of the two points, the three-dimensional coordinates (x, y, z) are parallel to the z-axis in one of the distance measurement images. need to move. zn is calculated from Equation (23), where the z-coordinate after translation is denoted as zn and the z-coordinate before translation is denoted as zb.
zn=zb-(lengthA-lengthB) (23)

In equation (23), lengthA is the subject distance in the distance measurement image that is not translated, and lengthB is the subject distance in the distance measurement image that is translated. For the subject distance, distance information to the focus position can be measured by, for example, a laser rangefinder. For example, there is a method of acquiring the subject distance for each image for all of a plurality of captured images. Further, there is a method of obtaining distance information (subject distance) from an arbitrary number of images, and a method of calculating the subject distance from the ratio of the focal length at the time of shooting and the lateral magnification. This saves time and effort compared to measuring the object distance for all of the multiple images. Specifically, the ratio of the lateral magnifications is D1:D2, and the distance from the focal length f to the object is denoted by D. The distance D can be calculated by Equation (24).
D = (D1/D2) x f (24)

The focal length f can be obtained by reading the lens state information at the time of photographing by the imaging device. Since the subject distance lengthA is the sum of the distance D and the focal length f, it can be calculated by Equation (25).
length A = D + f (25)
Let the three-dimensional coordinates of the ranging point A be (xa, ya, za), the three-dimensional coordinates of the ranging point B be (xb, yb, zb), and the line segment from the ranging point A to the ranging point B The length is denoted as LEN. LEN is calculated by Equation (26).
LEN
=√((xa-xb)^2+(ya-yb)^2+(za-zb)^2) (26)
"^" represents exponentiation.

As described above, the three-dimensional coordinates of each ranging point and the length between the two points can be calculated from the image for ranging determined based on the evaluation value, and the display unit 110 displays the distance between the two points, which is the measurement result. Show length.

FIG. 17 is a schematic diagram showing an example of a screen when presenting the measurement results to the user. The display screen 1001 shown in FIG. 17A displays the measured distance between the two points closest to the subject in the depth direction, and the display screen 1002 shown in FIG. 17B displays the depth of the subject. The measured distance between two points on the side in direction is indicated. In these examples, the position of two specified points and the length between the two points are displayed on the screen. The user can easily grasp the calculated distance between the distance measuring points.
In this embodiment, an image to be used for calculating the length between two points in a three-dimensional space is automatically selected based on the evaluation value at each ranging point for each ranging image, resulting in higher accuracy. distance measurement value can be calculated and presented.

[Second embodiment]
Next, a second embodiment of the invention will be described. In measuring the length in the depth direction using the split-pupil imaging method, the distance measurement accuracy varies depending on the amount of defocus, the brightness, contrast, SNR (signal-to-noise ratio) of the captured image, and the distance from the imaging device to the subject. to be influenced. Furthermore, since the depth of field is increased by setting the F-number to a small aperture and shortening the base line length, the defocus amount at the distance from the focus position can be reduced. For this reason, although the distance measurement range can be lengthened, the distance measurement resolution is lowered, and the distance measurement accuracy is lowered. For this reason, in order to more accurately measure the length in the three-dimensional space between two points specified for the subject, it is necessary to change the focus position with the same angle of view for the same subject. It is necessary to perform focus bracketing, in which multiple images are captured.

When performing focus bracket photography (hereinafter also referred to simply as bracket photography), the number of brackets (number of shots) can be set as many as the focus position can take, but such a method is not realistic due to time constraints. is not. Therefore, it is necessary to determine an additional photographing method based on the range of distance in the depth direction that can be obtained by one photographing at a certain focus position (hereinafter referred to as the effective distance measurement range). In this embodiment, processing for determining the shooting method for bracket shooting based on the effective distance measurement range will be described. Differences from the first embodiment will be mainly described, and detailed descriptions thereof will be omitted by using the same reference numerals as those of the first embodiment.

FIG. 18 is a block diagram showing a configuration example of the image measurement device 100 of this embodiment. Differences from FIG. 1 are as follows.
- An imaging unit 1102 is provided in place of the image input unit 102 .
- A parameter setting unit 1104, an effective ranging range calculation unit 1107, an additional imaging method determination unit 1108, and an imaging control unit 1109 are provided.
Other components (see reference numerals 108 to 110 in FIG. 1) are the same as those in the first embodiment, so illustration is omitted.

The imaging unit 1102 has a split-pupil imaging device, images a subject, and outputs captured image data to the memory 103 in accordance with a control command from the control unit 101 . The parameter setting unit 1104 performs setting processing of each parameter information input to the image measurement apparatus 100 by the user, and outputs setting data to the memory 103 according to control commands from the control unit 101 .

The photographing information at the time of photographing associated with the photographed image is obtained by the photographing information obtaining unit 104 and output to the memory 103 according to the control command from the control unit 101 . A defocus amount calculation unit 107 reads the image information and shooting information stored in the memory 103, and calculates the corresponding defocus amount for each coordinate of the image. The calculated defocus amount is stored in memory 103 according to a control command from control unit 101 .

An effective ranging range calculation unit 1107 reads image information, parameter information, shooting information, and defocus amount data stored in the memory 103, and calculates an effective ranging range of the image. The effective ranging range calculation unit 1107 performs effective ranging range determination processing and stores data of the effective ranging range in the memory 103 . An additional shooting method determination unit 1108 reads out the data of the defocus amount and the effective ranging range stored in the memory 103 and determines the additional shooting method. Data on the determined additional imaging method is stored in the memory 103 according to a control command from the control unit 101 . The imaging control unit 1109 reads the data of the additional imaging method stored in the memory 103 and controls the imaging unit 1102 via the control unit 101 .

The operation of the image measurement device 100 will be described in detail with reference to the flowchart of FIG. First, in S<b>1901 , the parameter setting unit 1104 performs parameter setting processing for the image measuring apparatus 100 . The parameters to be set include at least the following information.
(A) Shooting conditions (F number, ISO sensitivity, shutter speed, etc.) at the time of provisional distance measurement.
(B) Required ranging accuracy.
(C) Upper limit of the number of bracket shots.
Based on the set parameters, it is possible to determine shooting conditions for provisional distance measurement, and to determine an additional shooting method based on the results, the required distance measurement accuracy, and the upper limit of the number of bracket shots.

In S1902, provisional ranging processing is executed. A method of provisional distance measurement will be described later with reference to FIG. 20 . In S1903, the bracket shooting method is determined based on the result of the temporary distance measurement calculated in S1902. The bracketing method will be described later with reference to FIG. 21 . In S1904, bracketing is performed by the method determined in S1903. Then, the series of processing ends. Each process will be specifically described below.

FIG. 20 is a flow chart for explaining the temporary distance measurement method in S1902 of FIG. In S2001, the imaging unit 1102 images the subject based on the parameters set in S1901 of FIG. In S2002, the imaging information acquisition unit 104 acquires imaging information at the time of imaging performed in S2001. The imaging information is, for example, information (a) to (h) shown in the first embodiment. The focal length and lateral magnification ratio of the imaging optical system are acquired as focal position information of the imaging device.

In S2003, the defocus amount calculation unit 107 calculates the defocus amount of the image captured in S2001 based on the shooting information acquired in S2002. The details of the defocus amount calculation process are as described in the first embodiment. The defocus amount of the image captured for provisional distance measurement is calculated by repeating the calculation while shifting the position of the pixel of interest by one pixel.

In S2004, the control unit 101 calculates actual distance information from the defocus amount calculated in S2003. Denoting the actual distance at each point in the image as z, it can be calculated from equations (27) and (28).
dist=1/(1/(dist_d+DEF)-1/f) (27)
z=length−dist (28)
The meaning of each symbol is as follows.
dist: distance from the object principal point of the lens of the imaging unit 1102 to the object dist_d: distance from the image principal point of the lens of the imaging unit 1102 to the imaging sensor surface f: focal length of the imaging optical system length: image measurement The distance from the imaging sensor plane of the device 100 to the subject.

As for the distance length, for example, there is a method of measuring the distance to the focus position using a laser distance measuring unit, and a method of calculating from the ratio between the focal length f and the lateral magnification at the time of photographing. From the lateral magnification ratio D1:D2 and the focal length f, the distance D from the position of the focal length f to the object can be obtained by the above equation (24). The focal length f can be obtained from focal position information at the time of shooting. Since the distance length is the sum of the distance D and the focal length f, it can be obtained by Equation (29).
length = D + f (29)
After S2004, the process proceeds to return processing.

Next, focus bracket photography according to this embodiment will be described. FIG. 21 is a flow chart for explaining the bracketing method. In S2101, the effective ranging range calculation unit 1107 determines the range of distance to be obtained. A method for determining the bracket range will be described with reference to FIG.

FIG. 22 is a histogram of the results of the provisional distance measurement acquired in S1902 of FIG. 19 as a distance distribution in the image. The horizontal axis represents the distance, and the vertical axis represents the distribution of the distance. A distance distribution 2201 indicates the distance distribution amount of the image portion determined to be the background. The range of the distance distribution located in front of the distance distribution 2201 (on the imaging unit 1102 side) is determined as the bracket range (focus bracket imaging range). In addition to this method, in a device provided with a subject recognition unit that recognizes a face area or the like, the bracket range can be determined by further combining subject recognition results. For example, based on a histogram of subject distances recognized by the subject recognition unit, the distance range of the recognized subject is determined as the bracket range. Alternatively, a range having a distance distribution amount equal to or greater than a predetermined threshold can be determined as the bracket range. The number of bracketed ranges is not limited to one, and may be multiple ranges.

In S2102 of FIG. 21, the reliability of the image data acquired during provisional distance measurement is calculated. Confidence can be calculated by evaluating the luminance value, contrast, noise estimate, and distance to the focus position of the image. As for the brightness value of the image, the brightness average value is calculated by calculating the average value from the brightness values of the neighboring pixels at each coordinate of the image captured during the provisional distance measurement. The reliability of reproducibility based on the evaluation value LMR of the luminance mean value has already been described with reference to FIG. 15(C). As for the variance value related to the contrast, the variance value of neighboring pixels at each coordinate of the image captured during the provisional distance measurement is calculated. The reliability of reproducibility based on the evaluation value CNR of the variance has already been described with reference to FIG. 15(D).

FIG. 23A is a diagram for explaining evaluation values relating to the noise estimation amount of an image. The horizontal axis represents the noise estimator, and the vertical axis represents the evaluation value (denoted as NIS) of the noise estimator. The noise estimation amount is calculated based on the ISO value at the time of shooting. The noise estimator increases as the ISO value increases. If the noise estimation amount is large, the contrast becomes high, which affects the evaluation value of the variance value. Also, if noise affects the symmetry of the A image and the B image when calculating the defocus amount, there is a possibility that the distance measurement accuracy will decrease.

As shown in FIG. 23A, the evaluation value NIS is determined to decrease as the noise estimation amount increases. Denoting the noise estimation amount as nz, the evaluation value NIS is calculated by the following equation.
NIS = g1 - g2 x nz (30)
Both constants g1 and g2 are positive values and can be set arbitrarily. If we do not evaluate the noise estimator nz,
NIS=g3 (31)
and g3>0, and the value of g3 can be set arbitrarily.
As described above, as the noise estimator nz increases, the noise estimator evaluation value NIS decreases. As a result, it is possible to reduce measurement errors caused by noise, so that it is possible to calculate a more accurate distance measurement value.

FIG. 23B is a diagram for explaining evaluation values relating to the distance to the focus position. The horizontal axis represents the distance pn to the focus position, and the vertical axis represents the evaluation value DIS. As for the distance pn, the accuracy of ranging decreases as the distance increases. Ranging accuracy decreases in inverse proportion to the square of the distance pn. As shown in FIG. 23B, the evaluation value DIS is determined to decrease as the distance pn increases. The evaluation value DIS is calculated from the following formula using the distance pn.
DIS = h1 - h2 x pn (32)
Both constants h1 and h2 are positive values and can be set arbitrarily. If we do not evaluate the distance pn, then
DIS=h3 (33)
and h3>0, and the value of h3 can be set arbitrarily.
As described above, the evaluation value DIS decreases as the distance pn increases. As a result, it is possible to reduce the measurement error caused by the distance to the focus position, so that it is possible to calculate a more accurate distance measurement value.

In the reliability determination process, the reliability Rp is calculated from the product of each evaluation value, as in the first embodiment.
Rp = LMR x CNR x NIS x DIS (34)
Reliability Rp is normalized and expressed in the range of 0-1 by setting parameters so that the range of each evaluation value of LMR, CNR, NIS, and DIS is 0-1. As for each evaluation value and reliability, there is a calculation method using a parameter table in addition to a calculation method using a formula.

The reliability calculated for each pixel is finally converted to the reliability for each distance. The reliability by distance is calculated by averaging the reliability of a group of pixels having the same distance information calculated for each pixel or having close values of distance information. This makes it possible to calculate the reliability of pixels having specific distance information.

In S2103 of FIG. 21, the effective ranging range calculation unit 1107 calculates the effective ranging range. A specific example will be described with reference to FIG. In the present embodiment, a distance range in the depth direction that can be obtained in one shot at a specific focus position is calculated as the effective distance measurement range.

FIG. 24A is a schematic diagram illustrating the relationship between the distance from the focus position on the horizontal axis and the distance measurement accuracy on the vertical axis. An effective ranging range calculation unit 1107 performs effective ranging range determination processing based on the required ranging accuracy. A function curve 801 is determined as in Equation (35).
y=A×x^2+C (35)
y indicates a distance measurement accuracy value, and A indicates a coefficient determined from a conversion coefficient corresponding to the magnitude of the open angle of the center of gravity of the light flux passing through the pair of distance measurement pupil regions. x indicates the distance from the focus position, and C indicates the intercept calculated from the reliability calculated in S2102. The value of coefficient A is set so that the value increases as the F-number of the imaging optical system decreases. Regarding the intercept C, from the relationship between the highest reliability intercept (denoted as C1), the lowest reliability intercept (denoted as C2), and the reliability (denoted as R), Equation (36) is calculated as
C=C2-(C2-C1)×R (36)

For the function curve 801 defined in this way, the effective ranging range can be calculated based on the required ranging accuracy specified by the user. In other words, the effective ranging range is the range when the ranging accuracy (y value) is equal to or less than the designated ranging accuracy value.

FIG. 24(B) shows another function curve 802 for the function curve 801 illustrated in FIG. 24(A). A function curve 802 is an example of a function curve when the F value is large and the reliability is low. It is assumed that the required ranging accuracy in FIG. 24(A) is the same ranging accuracy in FIG. 24(B). In this case, it can be seen that the effective ranging range in the function curve 802 is narrower than the effective ranging range shown in FIG. 24(A). In this way, it is possible to calculate an effective ranging range that satisfies the required ranging accuracy based on the F-number and reliability.

In S2104 of FIG. 21, a process of dividing the bracket range by the effective ranging range calculated in S2103 is executed. A specific description will be given with reference to FIG. In FIG. 25A, the horizontal axis represents the distance, and the vertical axis represents the reliability R. In FIG. This is an example in which the bracket range is evenly divided by the effective distance measurement range when the reliability R is the maximum value.

An effective range-finding range 901a shown in FIG. 25A indicates the effective range-finding range when shooting is performed at the focus position 902a. When even division is performed, an area where the reliability R is evaluated may result in low ranging accuracy. A region of low ranging accuracy is indicated by a solid-line rectangular frame. A case where an area with low ranging accuracy occurs will be described with reference to FIG. 25(B).

FIG. 25B illustrates the relationship between the distance from the focus position shown on the horizontal axis and the ranging accuracy shown on the vertical axis. A function curve 901b is a function curve when the reliability R is the highest value. A function curve 902b is a function curve when reliability is considered, and is positioned above the function curve 901b. Comparing the first effective ranging range calculated from the function curve 901b and the second effective ranging range calculated from the function curve 902b, the first effective ranging range is the second effective ranging range. It can be seen that it includes a range and an area with low ranging accuracy. That is, the second effective ranging range is narrower. By evaluating the reliability, the effective distance measurement range is narrowed even if the focus position is the same.

In the example shown in FIG. 25A, if sufficient distance measurement accuracy cannot be obtained with uniform division, further division processing is performed. For example, ranges 903a, 904a, and 905a indicate effective ranging ranges when the reliability R is set to the maximum value, in which sufficient ranging accuracy cannot be obtained. These ranges are further divided.

FIG. 26A shows the result of equally dividing the effective range-finding range including the area with low range-finding accuracy in FIG. 25A into two. In this example, when the reliability R is further evaluated for the divided areas, there are areas with low ranging accuracy. Therefore, the area is divided into two equal parts, and the division process is repeated until there is no area with low distance measurement accuracy. Ultimately, the result of dividing the bracketed range is as shown in FIG.

FIG. 26(C) shows the focus positions 901e to 911e when the bracket range is divided by the effective ranging range as shown in FIG. 26(B), and the reliability for each distance. The required distance measurement accuracy can be obtained by photographing at each focal position after division. The above processing is performed by the additional imaging method determination unit 1108 . It is possible to improve the accuracy of ranging by dividing the bracket range into the effective ranging range. There is In this case, the number of divisions is limited based on the number of brackets set by the parameter setting section 1104 . Although it becomes impossible to obtain the necessary distance measurement accuracy for the entire bracket range, it is possible to set an efficient bracket photography method within the limited number of brackets.

Furthermore, there are cases where it is desired to give priority to the number of brackets when the bracket range becomes very wide. By setting the equations (35) and (36) so that the bracket range can be acquired as wide as possible with the limited number of brackets, the depth of field is deepened and bracket photography is performed. On the other hand, when it is desired to give priority to distance measurement accuracy when the bracket range is very narrow, bracket photography is performed with a shallow depth of field.

In this embodiment, the division processing is further performed to solve the problem that the required distance measurement accuracy cannot be satisfied by the equal division of the bracket range, and an area with low distance measurement accuracy is generated. In this case, if the interval is excessively reduced by finely chopping the focus positions without evaluating the reliability, the effective range-finding ranges overlap with each other, which is inefficient. In addition, the number of bracket shots becomes enormous. Therefore, the imaging control unit 1109 performs control so that bracket imaging is performed at the focus position shown in FIG. 26C based on the reliability evaluation result. It satisfies the required distance measurement accuracy and enables efficient bracket photography. According to this embodiment, it is possible to efficiently acquire distance information that satisfies arbitrary distance measurement accuracy conditions by determining the photographing method for bracket photography based on the effective distance measurement range whose reliability is evaluated.

In the above embodiment, as distance information corresponding to a plurality of image data, distance information corresponding to each distance measurement point is calculated from the image shift amount or defocus amount of image data pairs from different viewpoints, and the distance information is used to , the example of calculating the length between the specified distance measurement points has been described. Not limited to this, based on the image data pair at each focus position for each region of the image, that is, the image shift amount of the parallax image data pair, which parallax image data pair is used to use the distance information is determined in advance. It can be applied to the embodiment to be determined. In this case, the distance information is calculated only once for each area of the image, which is suitable for simplification and speeding up of processing.

[Other embodiments]
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

REFERENCE SIGNS LIST 100 Image measurement device 101 Control unit 102 Image input unit 104 Shooting information acquisition unit 105 Focus point selection unit 106 Corresponding point search unit 107 Defocus amount calculation unit 108 Evaluation value determination unit 109 Inter-focus point distance calculation unit

Claims (22)

a first acquisition means for acquiring a plurality of image data captured at a plurality of focus positions and from different viewpoints;
a second acquisition means for acquiring an image shift amount or defocus amount of a subject using image data pairs from different viewpoints at each focus position;
designating means for designating a plurality of distance measuring points in the image data;
calculating first distance information corresponding to each distance measuring point specified by the specifying means from the image shift amount or the defocus amount acquired by the second acquiring means, and using the first distance information; , calculating means for calculating the length between the specified distance measuring points in a three-dimensional space including the depth direction of the image;
a reliability determining means for determining a reliability corresponding to each ranging point specified by the specifying means in the plurality of image data;
The calculation means calculates the length between the distance measurement points from the first distance information of the distance measurement points in the selected image data using the reliability determined by the reliability determination means. An image measurement device characterized by: 2. The image measuring apparatus according to claim 1, wherein said designating means designates said distance measuring point by operating means, or designates said distance measuring point by detecting characteristic points of said image. The reliability determination means is one of a focus shift amount between images, an image shift amount or defocus amount, a luminance value, or a pixel variance value calculated for the range-finding points in the plurality of image data. The image measuring apparatus according to claim 1 or 2, wherein the reliability is determined by the above. a first acquisition means for acquiring a plurality of image data captured at a plurality of focus positions and from different viewpoints;
a second obtaining means for obtaining second distance information of a subject by laser ranging;
designating means for designating a plurality of distance measuring points in the image data;
calculating the second distance information corresponding to each distance measuring point in the image data, and using the second distance information, between the specified distance measuring points in a three-dimensional space including the depth direction of the image; a calculating means for calculating the length;
reliability determining means for determining reliability corresponding to the distance measuring point in the plurality of image data;
The calculation means calculates the length between the distance measurement points from the second distance information of the distance measurement points in the selected image data using the reliability determined by the reliability determination means. An image measurement device characterized by: a first acquisition means for acquiring a plurality of image data captured at a plurality of focus positions and from different viewpoints;
a second acquisition means for acquiring third distance information of a subject calculated using the focal length of the imaging optical system and the ratio of the lateral magnification;
designating means for designating a plurality of distance measuring points in the image data;
calculating the third distance information corresponding to each distance measuring point in the image data, and using the third distance information, between the designated distance measuring points in a three-dimensional space including the depth direction of the image; a calculating means for calculating the length;
reliability determining means for determining reliability corresponding to the distance measuring point in the plurality of image data;
The calculation means calculates the length between the distance measurement points from the third distance information of the distance measurement points in the selected image data using the reliability determined by the reliability determination means. An image measurement device characterized by: Range determination means for determining an effective ranging range using a pair of image data corresponding to focus positions;
a photographing method determining means for determining a photographing method for additional photographing by the photographing means from fourth distance information indicating the distribution of distances in the image data and the effective ranging range;
a control means for controlling imaging by the imaging means according to the determined imaging method ,
The first acquisition means acquires a plurality of image data captured at the plurality of focus positions and at different viewpoints by imaging controlled by the control means.
The image measuring device according to any one of claims 1 to 3, characterized in that: The range determination means determines the effective ranging range using one or more of the luminance value, contrast, noise estimation amount of the image, or the distance from the imaging means to the focus position. The image measuring device according to claim 6. 8. The image measuring apparatus according to claim 6 , wherein said photographing method determining means determines said photographing method using a histogram of said fourth distance information . 9. The image measurement method according to claim 8 , wherein said imaging method determining means determines, as an imaging range, a range of distance distribution located closer to said imaging means than a distance distribution corresponding to a background in said histogram. Device. further comprising recognition means for recognizing a subject in the image data;
9. The image measuring apparatus according to claim 8 , wherein said photographing method determining means determines a range of distance distribution of the subject recognized by said recognizing means in said histogram as a photographing range. 9. The image measuring apparatus according to claim 8 , wherein the photographing method determining means determines a range in which the distance distribution amount is equal to or greater than a threshold in the histogram as the photographing range. 8. The image measuring apparatus according to claim 6 , wherein said photographing method determining means determines a photographing method for performing a plurality of photographing operations by dividing a photographing range by said effective distance measurement range. 13. The image measuring apparatus according to claim 12 , wherein the imaging method determining means determines the imaging method by changing the depth of field. 14. The image measurement apparatus according to claim 13 , wherein the photographing method determination unit determines a photographing method that makes the depth of field deeper when giving priority to the number of shots compared to when giving priority to distance measurement accuracy. . The image measuring apparatus according to any one of claims 1 to 14 , further comprising display means for displaying the distance between the distance measuring points calculated by the calculating means. The image measuring device according to any one of claims 1 to 15 ;
an imaging device comprising: imaging means; The imaging means includes a plurality of microlenses and a plurality of photoelectric conversion units corresponding to each microlens,
The first obtaining means obtains the plurality of image data based on signals obtained by photoelectrically converting the light beams passing through different pupil partial regions of the imaging optical system and receiving them by the plurality of photoelectric conversion units. 17. The imaging device according to claim 16 . 18. The imaging apparatus according to claim 16 , wherein the calculating means outputs data of the calculated length between the distance measuring points to a display unit. An image measuring method executed by an image measuring device that acquires image data and performs measurement processing,
Acquiring a plurality of image data captured at a plurality of focus positions and from different viewpoints, and acquiring the image shift amount or defocus amount of the subject using the image data pairs from the different viewpoints at each focus position. process and
designating a plurality of ranging points in the image data;
determining a reliability corresponding to each designated ranging point in the plurality of image data;
First distance information corresponding to each designated distance measuring point is calculated from the image shift amount or the defocus amount, and the first distance information is used to calculate the designated distance in a three-dimensional space including the depth direction of the image. and a calculating step of calculating the length between the distance measuring points,
wherein, in the calculating step, a length between the distance measuring points is calculated from the first distance information of the distance measuring points in the selected image data using the determined reliability. measurement method. An image measuring method executed by an image measuring device that acquires image data and performs measurement processing,
a step of acquiring a plurality of image data captured at a plurality of focus positions and at different viewpoints, and acquiring second distance information of the subject by laser ranging;
designating a plurality of ranging points in the image data;
determining a reliability corresponding to the ranging point in the plurality of image data;
calculating the second distance information corresponding to each distance measuring point in the image data, and using the second distance information, between the specified distance measuring points in a three-dimensional space including the depth direction of the image; a calculating step of calculating the length,
In the calculating step, a length between the distance measuring points is calculated from the second distance information of the distance measuring points in the selected image data using the determined reliability. measurement method. An image measuring method executed by an image measuring device that acquires image data and performs measurement processing,
A plurality of image data captured at a plurality of focus positions and at different viewpoints are acquired , and third distance information of the subject calculated using the ratio of the focal length of the imaging optical system and the lateral magnification is acquired. process and
designating a plurality of ranging points in the image data;
determining a reliability corresponding to the ranging point in the plurality of image data;
calculating the third distance information corresponding to each distance measuring point in the image data, and using the third distance information, between the designated distance measuring points in a three-dimensional space including the depth direction of the image; a calculating step of calculating the length,
wherein, in the calculating step, a length between the distance measuring points is calculated from the third distance information of the distance measuring points in the selected image data using the determined reliability. measurement method. A program for causing a computer of an image measuring device to execute each step according to any one of claims 19 to 21 .

Images