JP7057717B2 - Phase difference detector, image processing device, and program - Google Patents

Description

INDUSTRIAL APPLICABILITY According to the present invention, a phase difference detector and an image processing device for detecting a phase difference indicating a deviation of the position of a subject from a set of images in the time direction or a spatial direction, and a computer for control in the image processing device can detect the phase difference. The present invention relates to a program for executing the evaluation value calculation according to the above.

There is a demand for finding a phase difference indicating a shift in the position of a subject that is commonly reflected in a plurality of very similar images that are changed only in a specific situation.

For example, by acquiring two images in succession in time with one image pickup camera and detecting the phase difference of the subject reflected in both of them from a set of images in the time direction, the image of the subject is displayed. It is possible to detect the movement speed of the camera and track the movement of the subject.

In addition, two imaging cameras with the same performance, such as a binocular camera, are spatially arranged and imaged simultaneously and in the same direction, and the phase difference of the subject reflected in both of them is obtained from the obtained pair of images in the spatial direction. By detecting, it is possible to detect the distance between the image pickup camera and the subject, or the spatial position between the subjects.

Furthermore, using the phase difference detection pixel that blocks a part of the pixel of the image sensor, it automatically determines whether the current in-focus position is closer to or far from the subject from the way the "blurring" spreads when out of focus. Focusing techniques are also known (see, for example, Patent Document 1). At that time, the phase difference detection pixels whose light-shielding portions are different (generally symmetrical in the left-right and up-down directions) from one image captured by one image pickup camera equipped with an image sensor having the phase difference detection pixel. A set of phase difference images is formed by extracting and accumulating pixel values adjacent to each of the light-shielding portions, and the phase difference of the set of phase difference images is detected to realize the determination of the in-focus position.

By the way, in the technique of detecting the phase difference indicating the deviation of the position of the subject from a set of images in the temporal direction or the spatial direction, the following formula (number) is used as a technique for obtaining an evaluation value for detecting the phase difference. As shown in 1), for example, the difference is taken while shifting the right-eye image and the left-eye image constituting a set of phase difference images one pixel at a time to the left and right, and the evaluation value (correlation value) by the sum of the absolute values of the differences is minimized. The technique of finding the shift width k is common. Here, SAD (Sum of Absolute Difference; absolute value of difference) showing a correlation value is described as an example as an evaluation value, but SSD (Sum of Squared Difference; squared value of difference) may be used as an example.

L and R in the above equation (Equation 1) represent the pixel values of the left-eye image and the right-eye image, respectively, and the pixel addresses are indicated by the subscripts at the lower right of L and R, respectively. For example, when the SAD value of the in-focus position detection region is obtained from the address values of i and j indicating the pixel positions of the pixel portions in the image pickup element, Li _{+ k, j} and Ri, _j in the above equation (Equation 1). Represents the j-th column i + k-th pixel value of the phase difference detection pixels (phase difference detection pixels L, R) of a pair of diagonal two pixels discretely arranged in the pixel portion, and J is used for correlation calculation. The number of pixels to be used and I are the numbers in the column direction of the pair of images for which the correlation calculation is performed, and J and I are appropriately set according to the distance measuring field length (distance measuring field length) and the like. K obtained from the minimum SAD value indicates the shift width of the phase difference image (phase difference between the right eye image and the left eye image).

Therefore, the evaluation value based on the above equation (Equation 1) can be used for detecting the in-focus position, detecting the amount of movement of the subject on the image, and the like.

Although the pixel value is directly used in the above equation (Equation 1), the color histogram of the region of interest on the image can be used as the image feature amount, or the shape of the subject can be differentiated on the image plane. There is also a technique for extracting as an image feature amount. In particular, when comparing images that are continuous in time, it is possible to predict the position of the subject on the image at the next moment by using the image feature amount based on the pixel value related to the movement of the subject in the immediately preceding time. can.

On the other hand, in various manufacturing devices such as semiconductors and product inspection devices, the correlation value between the two patterns to be collated is calculated while shifting the relative positional relationship between the image pattern of the registered image and the image pattern of the collated image. , A technique for measuring the positional deviation of a subject in both images by finding the position where the correlation value is maximized is known (see, for example, Patent Document 2). In particular, in the technique of Patent Document 2, a plurality of types of complex filters are applied to two patterns to be collated, and pattern matching is performed based on a feature amount to which not only pixel value information but also phase information is added. It enables accurate position shift measurement.

Japanese Unexamined Patent Publication No. 2018-022128 Japanese Patent No. 5558127

As described above, as a technique for detecting the phase difference of a conventional image, a technique using an evaluation value based on the above equation (Equation 1), a color histogram of a region of interest on an image, and a pixel value are used as an image feature amount. Techniques for There is a technique to make a feature quantity with phase information added instead.

However, since the technique using the evaluation value based on the above equation (Equation 1) is a phase difference detection technique based only on the pixel value information, the number of significant digits of the shift width k of the phase difference image is also limited. Therefore, there is room for improvement in detection accuracy.

In addition, for example, in the technique using a color histogram, an image is taken in an environment with little color information, such as an image taken in a dark place, an image taken with an infrared camera, or an image taken in an environment where the influence of light of a specific frequency is dominant such as in the evening. The detection accuracy of the phase difference deteriorates for the captured image.

In addition, for example, the technique of differentiating on an image plane to extract the shape characteristics of a subject is a case where the phase difference becomes one pixel or less because the amount of movement of the subject is small or the resolution of the image pickup camera is low. Cannot accurately detect the phase difference. On the contrary, if the amount of pixel shift corresponding to the phase difference becomes large due to a large amount of movement of the subject or a high resolution of the image pickup camera, the pixels in the range (scanning range) for detecting the subject. There is a problem that it is necessary to take a large number and the amount of calculation becomes large.

Further, the technique disclosed in Patent Document 2 can improve the detection accuracy of the phase difference by adding the feature amount to which not only the pixel value information but also the phase information is added, but it is multiplied by a plurality of types of complex filters. Therefore, complicated conditional branching and calculation are required, and the processing cost itself increases, which causes problems such as an increase in device cost.

However, all conventional techniques for detecting the phase difference of an image have advantages and disadvantages. Therefore, a technique for improving the detection accuracy of the phase difference at a low processing cost without causing a difference in the detection accuracy of the phase difference regardless of the resolution of the image pickup camera and in an environment where there is little color information. Is desired. For example, in an imaging camera that is expected to be used in various environments such as a television camera, the shift width k of the phase difference image based on the evaluation value based on the above equation (Equation 1) is effective at low processing cost regardless of the imaging environment. There is a demand for a technique for improving the detection accuracy of the phase difference based on the evaluation value of the number of significant digits higher than the number of digits.

In view of the above-mentioned problems, an object of the present invention is to provide a phase difference detector, an image processing device, and a program that stably detect a phase difference with high accuracy regardless of a shooting environment at a low processing cost.

That is, the phase difference detector of the present invention is a phase difference detector that detects a phase difference indicating a deviation of the position of a subject from a set of images in the time direction or the spatial direction, and each attention of the set of images. The main frequency detecting means for detecting the wavelength of the main frequency of the image signal in the region and the image signal in each region of interest of the set of images are set as the processing target image, and the processing target image is complex based on the wavelength. A CREW calculation means that performs rectangular wavelet transform (CREW) to generate a signal vector of a complex frequency component for an image signal in each region of interest in the set of images, and a signal vector in each region of interest in the set of images. A phase difference calculating means for calculating the phase difference by comparison and a coordinate calculating means for calculating the displacement coordinates of the subject in at least one of the attention regions of the set of images based on the calculated phase difference are provided. It is characterized by that.

Further, in the phase difference detector of the present invention, the basis of the complex rectangular wavelet transform is represented by a filter of a complex rectangular sequence having element values of at least four quadrants.

Further, in the phase difference detector of the present invention, the phase difference calculating means is characterized in that the phase difference is calculated by the Manhattan distance of each signal vector mapped on the complex plane.

Further, in the phase difference detector of the present invention, the CREW calculation means spatially divides and compresses the image to be processed in at least one direction based on the wavelength, and the number of pixels in the one direction is N (N). It is characterized in that a compressed image with ≧ 4) is generated, and a complex rectangular wavelet transform (CREW) is performed on the compressed image.

Further, in the phase difference detector of the present invention, the CREW calculation means obtains element values of four quadrants with respect to a compressed image having N (N ≧ 4) pixels in one direction divided and compressed at 1/4 of the wavelength. It is characterized by performing a complex rectangular wavelet transform based on the CREW filter having.

Further, in the phase difference detector of the present invention, the CREW calculation means obtains element values of four quadrants with respect to a compressed image having N (N> 4) pixels in one direction divided and compressed at 1/4 of the wavelength. It is characterized in that a complex rectangular wavelet transform based on the CREW filter is performed to generate a plurality of types of signal vectors according to the pixel addresses.

Further, in the phase difference detector of the present invention, the CREW calculation means directly connects each element value of the complex rectangular wavelet transform to the processing target image by 1/4 of the wavelength to make a CREW filter. It is characterized in that the CREW calculation is performed by making the above redundant.

Further, in the phase difference detector of the present invention, the CREW calculation means is a one-dimensional CREW filter spatially targeted in one direction and a two-dimensional spatially targeted two-dimensional as the complex rectangular wavelet transform. It is characterized in that the wavelet transform is performed based on the CREW filter of No. 1 or a CREW filter having three or more dimensions including the time direction in addition to the two directions.

Further, in the phase difference detector of the present invention, the phase difference detector is configured to sequentially detect the phase difference from images continuous in the time direction, and the main frequency detecting means sequentially detects the phase difference. When the power of the signal vector generated by the means for calculating the wavelength and the CREW calculation means is monitored as an initial value at the time of the detection, and the power of the signal vector is out of a predetermined range, the wavelength It is characterized by having a means for calculating again.

Further, in the phase difference detector of the present invention, the phase difference detector is configured to sequentially detect the phase difference from images continuously in the time direction, and the CREW calculation means detects the phase difference in time. When performing continuously for images that are continuous in the direction, the signal vector is sequentially recalculated as the current value and compared with the signal vector of the previous value, and the difference between the signal vector of the current value and the signal vector of the previous value. Is not less than or equal to the threshold, and if it is less than or equal to the threshold, the signal vector of the current value is updated as a reference signal vector to be compared in the next detection of the phase difference. However, the main frequency detecting means is characterized by having a process of recalculating the wavelength.

Further, the image processing apparatus according to the present invention is the phase difference detector of the present invention configured to sequentially detect the phase difference from images continuously in the time direction, and the subject based on the phase difference. It is characterized by comprising a signal processing means for detecting a movement speed on an image, tracking a movement of a subject, or detecting a dynamic change of a subject.

Further, the image processing apparatus of one aspect according to the present invention includes the phase difference detector of the present invention configured to sequentially detect the phase difference from a set of images simultaneously and simultaneously imaged in the spatial direction. It is characterized by comprising a signal processing means for performing a physical distance measurement process relating to a subject based on the phase difference.

Further, the program of the present invention is configured as a program for causing the computer to function as the phase difference detector of the present invention.

According to the present invention, the phase difference can be detected stably and with high accuracy regardless of the shooting environment at low processing cost. In particular, according to the present invention, the evaluation value of the number of significant digits higher than the number of significant digits of the shift width k of the phase difference image based on the evaluation value based on the above equation (Equation 1) can be set, and the phase difference detection can be performed. High accuracy and low mounting cost can be achieved.

It is a block diagram which shows the schematic structure of the phase difference detector of 1st Embodiment by this invention, and the image processing apparatus provided with this. FIGS. (A) to (e) are diagrams showing processing examples of the main frequency detection unit in the phase difference detector of the first embodiment, respectively. (A) to (c) are diagrams showing a processing example of a complex rectangular wavelet transform (CReW: Complex Rectangular Wavelet Transform) calculation unit of one embodiment (Type 1) in the phase difference detector of the first embodiment, respectively. FIGS. (A) to (C) are diagrams showing processing examples of the complex rectangular wavelet transform (CReW) calculation unit of the first embodiment (Type 2) in the phase difference detector of the first embodiment, respectively. (A) to (c) are diagrams showing a processing example of the Fourier transform of the comparative example, respectively. (A) and (b) compare the processing cost of the complex rectangular wavelet transform (CReW) calculation unit of the first embodiment (Type 1 and 2) and the Fourier transform of the comparative example in the phase difference detector of the first embodiment, respectively. It is a figure which shows as possible. (A) to (c) are diagrams showing more specific processing examples of the complex rectangular wavelet transform (CREW) calculation unit in the phase difference detector of the first embodiment according to the present invention, respectively. It is a flowchart which shows the processing of the image processing apparatus provided with the phase difference detector and the signal processing unit of 1st Embodiment by this invention. It is a figure which shows the operation example of the image processing apparatus provided with the phase difference detector of 1st Embodiment by this invention. It is a figure which shows the operation of the image processing apparatus provided with the phase difference detector of 1st Embodiment by this invention, and the operation | effect thereof. It is a block diagram which shows the schematic structure of the phase difference detector of 2nd Embodiment by this invention, and the image processing apparatus provided with this. It is a flowchart which shows the processing of the image processing apparatus provided with the phase difference detector and the signal processing unit of 2nd Embodiment by this invention. It is a figure which shows the operation example of the image processing apparatus provided with the phase difference detector of 2nd Embodiment by this invention.

Hereinafter, with reference to the drawings, the phase difference detector 11 of each embodiment according to the present invention and the image processing device 1 including the same will be described.

[First Embodiment]
FIG. 1 is a block diagram showing a schematic configuration of a phase difference detector 11 according to a first embodiment of the present invention and an image processing device 1 including the same. The image processing device 1 sequentially inputs the captured images acquired by one image pickup camera 10 continuously in time, and is continuously acquired by the phase difference detector 11 in time. It is configured as a device for detecting the movement speed of a subject on the image and tracking the movement of the subject by detecting the phase difference of the subject reflected on both of them based on a set of captured images.

(Image camera)
First, the image pickup camera 10 according to the present embodiment is a camera that continuously images a subject in a time-sequential manner to form and output an image captured image, and includes an image pickup lens 101, an image pickup element 102, and an image processing unit 103. Be prepared.

The image pickup lens 101 according to the present embodiment is an optical lens for forming an image of a subject on the image pickup element 102, and may be driven so as to be able to automatically focus. However, here, a state in which the focus is on a desired subject is set. Let's take an example.

The image pickup device 102 according to the present embodiment has an image pickup unit composed of a plurality of pixels, and has phase difference detection pixels having different light-shielding portions (generally symmetrical in the left-right and up-down directions) in the image pickup unit. However, here, an example of not having such a phase difference detection pixel will be described.

The image processing unit 103 according to the present embodiment sequentially forms an image captured by a subject obtained from the image sensor 102 in frame units, and outputs the image to the signal processing unit 13 and the phase difference detector 11. In particular, the image processing unit 103 forms an image pickup image P1 in which the subject obtained from the image pickup element 102 is captured at the time when the operation of the image phase difference detector 11 starts, and outputs the image processing unit 103 to the signal processing unit 13 and the phase difference detector 11. do.

After the operation of the image phase difference detector 11 is started, the image processing unit 103 of the present embodiment continuously captures the subject obtained from the image pickup device 102 in time after the formation of the image pickup image P1. Is formed and output to the signal processing unit 13 and the phase difference detector 11.

In this way, the image processing unit 103 of the present embodiment sequentially forms an image captured by the subject obtained from the image sensor 102 in frame units and outputs the image to the signal processing unit 13. Then, at the time when the operation of the image phase difference detector 11 is started, the image processing unit 103 of the present embodiment sequentially receives the signal processing unit 13 for the pair of captured images P1 and P2 acquired continuously in time. And output to the phase difference detector 11.

(Image processing device)
The image processing device 1 includes a phase difference detector 11, a memory 12, and a signal processing unit 13.

The phase difference detector 11 initially sets the area of interest (regardless of vertical, horizontal, or diagonal) of the subject based on the instruction from the operation unit 16 and starts operation, and one image pickup camera 10 temporally starts the operation. It is a functional unit that detects the phase difference of the subject reflected on both of the continuously acquired two images P1 and P2. The initial setting of the area of interest of the subject can be specified by the operator at an arbitrary position, or can be set to a predetermined range such as the central area of the captured image. At the start of operation of the image phase difference detector 11, the initially set area of interest of the subject is dynamically calculated in coordinates by the subsequent operation of the image phase difference detector 11 to detect the moving speed of the subject on the image. It is cut out so as to track the movement of the subject or the subject, for example, by enclosing it in a square shape.

The memory 12 has a function of temporarily storing captured images P1 and P2 required for processing of the signal processing unit 13 while sequentially updating them, and a function of temporarily storing each signal value required for processing of the phase difference detector 11. ..

The signal processing unit 13 determines the phase difference φ of the subject reflected in both of the captured images P1 and P2, which are continuously acquired in time by the phase difference detector 11. It is a functional unit that cuts out a subject area from the captured image P2 using the corresponding displacement coordinates x, detects the moving speed of the subject on the image, and tracks the movement of the subject. Therefore, the signal processing unit 13 sequentially inputs the captured images acquired by one image pickup camera 10 in a timely manner, and a set of two images continuously acquired in the input time. The captured images are temporarily updated and temporarily stored in the memory 12. Then, for example, the signal processing unit 13 cuts out the subject area from the sequentially input captured images so as to surround the subject area, for example, in a square shape, and displays the section on the display unit 14 such as a display panel, or displays the cut out image area. Image recording is repeated for the recording unit 15 in the manner shown.

More specifically, the phase difference detector 11 and the image processing device 1 will be described in detail below.

(Phase difference detector)
As shown in FIG. 1, the phase difference detector 11 of the present embodiment includes a main frequency detection unit 111, a CREW calculation unit 112, a phase difference calculation unit 113, and a coordinate calculation unit 114.

The main frequency detection unit 111 operates so as to calculate the wavelength λ of the main frequency of the captured image P1 at the time when the operation of the image phase difference detector 11 starts. That is, the main frequency detection unit 111 detects the frequency component exemplified with reference to FIG. 2 for the region of interest of the captured image P1 input from the image processing unit 103 at the start of operation of the image phase difference detector 11. The wavelength λ of the main frequency at which the power (amplitude of the pixel value) of the captured image P1 is maximized is calculated, and the wavelength λ is temporarily stored in the memory 12.

Further, the main frequency detection unit 111 of the present embodiment monitors the power of the signal vector F of the frequency component calculated for each of the two image pickup images P1 and P2 in the region of interest by the CREW calculation unit 112 described later. , The wavelength of the main frequency when the power of the signal vector F of the frequency component is out of a predetermined range (absolute value or relative value determined based on the power at the time of the immediately preceding frequency detection). It operates to calculate λ again and update the temporary storage for the memory 12.

2 (a) and 2 (e) are diagrams showing processing examples of the main frequency detection unit 111 in the phase difference detector 11 of the first embodiment, respectively.

The main frequency detection unit 111 inputs the captured image P1 and detects the wavelength λ of the typical main frequency of the image signal in the region of interest. Therefore, first, as shown in FIG. 2A, the main frequency detection unit 111 of the image signal A length M shorter than the length is set, and a region of the image width M is extracted from the image signal having the maximum amplitude a. In FIG. 2, for convenience of explanation, a rectangular image signal having a maximum amplitude a is normalized and displayed, but the actual signal (pixel value) has various amplitudes of positive values.

Subsequently, as shown in FIGS. 2 (b) and 2 (c), the main frequency detection unit 111 divides the signal by the length of M / 2, and the first half portion divided by the image width M / 2. Extract the latter half and each.

Subsequently, as shown in FIG. 2D, the main frequency detection unit 111 superimposes the first half portion and the second half portion divided by the image width M / 2 in the signal order and makes a difference, resulting in a difference. Calculate the signal (length M / 2). As a result, a difference signal emphasized as the power P is obtained.

Subsequently, as shown in FIG. 2E, the main frequency detection unit 111 calculates the normalized power obtained by dividing the sum of the absolute values of the powers P of the difference signals by M.

Then, the main frequency detection unit 111 repeats while changing the value of M in a range shorter than the length of the image signal, obtains the value of M that maximizes the normalized power, and uses this value of M as the wavelength of the main frequency. Let it be λ.

That is, the technique illustrated in FIG. 2 emphasizes the power P by using the value obtained by subtracting the value of the autocorrelation function of the M / 2 shift from the total power of the signal, and detects the representative frequency. There is. Therefore, the difference signal shown in FIG. 2D is a signal having a power of a substantially square wave in which the power of the frequency component of the wavelength M / 2 and the frequency component of an integral multiple thereof is suppressed. The power of a square wave having an odd multiple frequency is also mixed, but the energy is also an odd number, so there is no problem in calculating the main frequency.

The main frequency detection unit 111 has a main frequency based on other techniques for detecting the main frequency, such as calculating a power spectrum using a Fourier transform, in addition to the technique based on the autocorrelation function according to FIG. It may be configured to detect the wavelength λ. For example, the main frequency detection unit 111 assumes that the subject is a range of similar colors defined by a predetermined standard based on the color information of the center of the input image signal, and sets the length twice as wide as the main frequency. The wavelength λ may be used.

The CREW calculation unit 112 first reads out coordinate calculation information indicating the wavelength λ, the number of divided blocks N, and the displacement coordinates x from the memory 12, and is a set of two images input from the image processing unit 112 based on the displacement coordinates x. The region of interest of the captured images P1 and P2 is calibrated, and the image signals of each of the region of interest of the two captured images P1 and P2 based on the wavelength λ are N blocks or more (N) in one direction at the wavelength λ / 4. By dividing into (a fixed integer value of ≧ 4) and calculating the sum (or average) of each, a compressed image having N pixels in the dividing direction is generated.

Since the value of the displacement coordinate x is not calculated at the time when the operation of the image phase difference detector 11 is started, the displacement coordinate x as the initial value indicates zero displacement amount. That is, at the time when the operation of the image phase difference detector 11 starts, the region of interest of the captured images P1 and P2 of the two images is at the same position as at the time of the initial setting, but after that, the images are continuously acquired 2 in time. When detecting the phase difference of a set of captured images, the region of interest of the captured image after the time is captured before the time by calibration based on the displacement coordinates x corresponding to the subject region to be cut out. The coordinates are different from the area of interest in the image. Further, the number of divided blocks N is a predetermined value stored in the memory 12 and can be externally set from the operation unit 16.

Then, the CREW calculation unit 112 has element values of at least four quadrants in the direction of the compressed image having N pixels in the division / compression direction in each of the attention regions of the two image capture images P1 and P2. Complex Rectangular Wavelet Transform (CReW) based on a complex rectangular wavelet (CREW filter) represented by a complex number sequence is applied, and one type of signal vector F for the frequency component of the complex (real number component and imaginary number component) is obtained. The above is generated and output to the phase difference calculation unit 113.

However, in the processing order of the time series, the signal vector F for the captured image P1 obtained by the CREW calculation unit 112 is temporarily stored in the memory 12 and used as the reference signal vector F'for phase difference detection, and is used as the reference signal vector F'for the phase difference detection. The signal vector F for the captured image P2 obtained by the above is output to the phase difference calculation unit 113.

Here, two examples of Type1 and Type2 will be described with respect to the complex rectangular wavelet transform (CReW) of one embodiment.

3 (a) to 3 (c) are diagrams showing processing examples of the CREW calculation unit 112 of one embodiment (Type 1) in the phase difference detector 11 of the first embodiment, respectively. Further, FIGS. 4 (a) to 4 (c) are diagrams showing processing examples of the CREW calculation unit 112 of one embodiment (Type 3) in the phase difference detector 11 of the first embodiment, respectively.

The CREW calculator 112 is essentially a wavelet transform with complex components, but constitutes a filter represented by a rectangular sequence with element values in four quadrants, thereby adding phase information for power and normalization. Strictly speaking, it differs from the general wavelet transform in that it converts into a signal vector that reduces the phase calculation cost.

For example, in FIGS. 3 (a) and 4 (a), a signal component having a period N (an integer fixed value of N ≧ 4) included in an image signal having a maximum amplitude a is taken as an example of application of the CREW filter. explain. In FIGS. 3A and 4A, for convenience of explanation, a rectangular image signal having a maximum amplitude a is normalized and displayed, but the actual signal (pixel value) has various positive values. Has an amplitude of.

In FIGS. 3A and 4A, the length of the signal (the number of pixels in one direction in the case of an image) to be the target of the signal vector calculation of the frequency component by the CREW calculation unit 112 is included in the signal. The normalized phase of the frequency component of the period N (the value obtained by dividing the phase θ of the frequency component by the phase of one cycle, and when θ ′ = 1, the signal is displaced by exactly one cycle of the signal cycle) is θ. ′, The power (maximum amplitude) is a. Therefore, the signal component in the (N / 2) region has a positive amplitude a, and is negative in the other (N / 2) regions {(θ'N) and (N / 2-θ'N) regions}. It can be represented by a signal component of amplitude a.

Then, the shape example of the CREW filter of Type 1 applied to the signal component shown in FIG. 3 (a) is shown in FIG. 3 (b), and the shape example of the CREW filter of Type 2 applied to the signal component shown in FIG. 4 (a) is shown. It is shown in 4 (b). Both Type1 and Type2 are represented by a rectangular sequence having four quadrant element values that divide the signal period N into four, and are CREW filters that are rectangular waves in which the real and imaginary components are staggered (orthogonal). ing. That is, in Type 1, the CREW filter is represented by a rectangular sequence [1, i, -1, −i] having element values of four quadrants when expressed as a real number component “1” and an imaginary number component “i”. Further, Type 2 is a CREW filter represented by a rectangular sequence [1-i, 1 + i, -1 + i, -1-i] having four quadrant element values. In FIGS. 3 (b) and 4 (b), the amplitude of the CREW filter is b.

Then, by applying the CREW filter of Type 1 shown in FIG. 3 (b) to the signal component shown in FIG. 3 (a) and performing signal analysis, the frequency is as shown on the complex plane in FIG. 3 (c). The value (complex number) that the component can take depending on its phase indicates the signal vector on the side of the square. In the case of Type 1 shown in FIG. 3 (c), one side is abN and each side is a square parallel to the real number axis or the imaginary number axis, respectively, with the origin as the center.

Further, by applying the CREW filter of Type 2 shown in FIG. 4 (b) to the signal component shown in FIG. 4 (a) and performing signal analysis, the frequency is as shown on the complex plane in FIG. 4 (c). The value (complex number) that the component can take depending on its phase indicates the signal vector on the side of the square. In the case of Type 2 shown in FIG. 4 (c), it is a square centered on the origin and surrounding abN and -abN on the real number axis and the imaginary number axis, respectively.

The phase difference φ of the signal vectors to be compared is obtained by the difference of the normalized phase θ ′ of each signal vector, and the difference of the normalized phase θ ′ of each signal vector mapped on the complex plane is calculated by the Manhattan distance. Can be calculated.

Generally, the amplitude b in the wavelet transform filter is determined according to the shape of the wavelet transform filter and the image resolution, based on the general wavelet transform rules. However, in the CREW filter according to the present invention, since the phase component is handled so as not to be affected by the value of b, b = 1 may be set in terms of mounting. In other words, by setting b = 1, the signal vector is displaced on the side of the square on the complex plane by the phase value, and there is an advantage that the processing cost is further reduced.

As comparative examples, FIGS. 5 (a) to 5 (c) show examples of Fourier transform processing. FIG. 5 (a) illustrates the same signal components as those in FIGS. 3 (a) and 4 (a) above. An example of the shape of the Fourier transform filter applied to the signal component shown in FIG. 5 (a) is shown in FIG. 5 (b). The Fourier transform sequence shown in FIG. 5 (b) is an example using a complex number e ^{-2πi α / N} having an amplitude b with respect to the variable α, and is a trigonometric function sequence [e ^{-2π i / N)} having element values in four quadrants. e ^{-4πi / N} , e ^{-6πi / N} , e ^{-8πi / N} ] is expressed as a sequence.

Then, by applying the Fourier transform filter shown in FIG. 5 (b) to the signal component shown in FIG. 5 (a) and performing signal analysis, the frequency is shown on the complex plane in FIG. 5 (c). As a value (complex number) that the component can take depending on its phase, it indicates a signal vector on the circumference. In the case of the example shown in FIG. 5 (c), it is a circle surrounded by a radius of 2abN / π on the real number axis and the imaginary number axis with the origin as the center. In this case as well, by treating it as b = 1 in terms of mounting, the signal vector scans the line segment indicated by the circle on the complex plane by the phase value, which will be described below with reference to FIG. As described above, from the viewpoint of processing cost, it is disadvantageous as compared with the CREW filter according to the present invention.

6 (a) and 6 (b) show the CreW calculation unit 112 of one embodiment (Types 1 and 2) in the phase difference detector 11 of the first embodiment, respectively, and the Fourier transform of the comparative example shown in FIG. It is a figure which shows the processing cost of the above in a comparable manner.

In particular, FIGS. 6 (a) and 6 (b) collectively illustrate the processing costs of the power P and the normalized phase θ'. In the normalized phase, the value of each coefficient changes depending on the positive / negative or magnitude of the real number component and the imaginary number component of the frequency component, respectively.

As shown in FIG. 6 (a), the CREW calculation unit 112 according to the present invention can calculate the power and the normalized phase corresponding to the complex number regardless of whether the CREW filter of Type 1 or 2 is used. It can be realized by several conditional branching (detection of the quadrant on the complex plane of the frequency component), absolute value calculation within several times according to the conditional branching, power of 2, and one division.

In particular, in arithmetic processing, absolute value calculation is realized by inversion and addition of bit register values. The power of 2 is realized by shifting the value of the register. As a whole, the CREW calculation unit 112 according to the present invention can be applied to a high-resolution image or a high-frequency video by using any of the Type 1 and Type 3 CReW filters, and the processing cost is reduced and high-speed processing is possible. Therefore, it is possible to suppress an increase in mounting cost.

On the other hand, as shown in FIG. 6B, in the Fourier transform of the comparative example, the absolute value is calculated within several times depending on the conditional branching (detection of the quadrant on the complex plane of the frequency component) and the conditional branching. It cannot be realized by a power of 2 and a power of 2, and a single division, and requires complicated operations. This is also the case when a general wavelet transform is used. Therefore, the CREW calculation unit 112 according to the present invention can reduce the processing cost by using the CREW filter as exemplified by Types 1 and 2.

In the sequence representation of the CREW filter illustrated by Type 1 and 2 shown in FIG. 6A and the like, when expressed as a real number component "1" and an imaginary number component "i", a rectangular sequence of rectangles having element values of four quadrants [1, i , -1, -i] or the rectangular sequence [1-i, 1 + i, -1 + i, -1-i] can be arbitrarily replaced in the element order or arrangement. For example, the same applies even if the rectangular sequence of Type 1 is set to [-1, i, 1, -i] or the rectangular sequence of Type 1 is set to [1 + i, 1-i, -1 + i, -1-i]. The processing result and effect of the above can be obtained.

7 (a) to 7 (c) are diagrams showing more specific processing examples of the CREW calculation unit 112 in the phase difference detector 11 of the first embodiment according to the present invention, respectively.

As shown in FIG. 7A, the CREW calculation unit 112 first reads out the coordinate calculation information indicating the wavelength λ, the number of divided blocks N, and the displacement coordinates x from the memory 12, and the image processing unit 112 is based on the displacement coordinates x. The area of interest of the pair of captured images P1 and P2 input from 112 is defined, and this area of interest is used as the image to be processed. Since the value of the displacement coordinate x is not calculated at the time when the operation of the image phase difference detector 11 is started, the displacement coordinate x as the initial value indicates zero displacement amount.

The image to be processed by the CREW calculation unit 112 is based on the attention area of the subject initially set based on the instruction from the operation unit 16 in this example in order to further reduce the processing cost. It may be the whole.

Then, the CREW calculation unit 112 sets each of the attention regions of the two image pickup images P1 and P2 in one direction at the wavelength λ / 4 based on the wavelength λ and the number of divided blocks N (horizontal direction in the illustrated example). ) Is divided into N or more blocks (N = 6 in the illustrated example), and the total sum (or average) of each block is calculated. When viewed as an image, a compressed image with 6 pixels, which is very coarse in one direction, is generated. At this time, the width of each block is one-fourth of the wavelength λ of the CREW filter used. With four blocks, the signal length is equivalent to the length of the CREW filter.

Subsequently, as shown in FIG. 7B, the CREW calculation unit 112 spatially moves (scans) the compressed image of four consecutive blocks out of the compressed images of six blocks in the one direction (division direction). Then apply the CREW filter. That is, for example, in the case of Type 1 shown in FIG. 3 (b), the CREW filter composed of the four elements [1, i, -1, −i] is applied to the compressed image of four blocks. Further, in the case of Type 2 shown in FIG. 4 (b), a CREW filter composed of four elements [1-i, 1 + i, -1 + i, -1-i] is applied to a compressed image of four blocks. In any case of the Type 1 and Type 2 CREW filters, a value in which the number of blocks is reduced by 3 is obtained.

Subsequently, as shown in FIG. 7 (c), the CREW calculation unit 112 converts the number of pixels in the vertical direction × 3 types of signal vectors (complex signals). As a result, three types of signal vectors corresponding to the addresses of the pixels consisting of three sets of real number components and three sets of imaginary number components are obtained, and ideally, the signal vectors are phase-rotated by 90 °. That is, if N = 4, one signal vector is obtained, if N = 5, two signal vectors are obtained, and if N = 6, three signal vectors are obtained. Therefore, the larger the value of N (the larger the number of signal vectors), the wider the range as the feature quantity expression for one processed image, but even if it is larger than necessary, the signal vector is unnecessary for the implementation purpose. Will be obtained and waste will occur as a processing cost. From the results of the experiment, it is known that it is preferable to set N = 6, but it is preferable to set it to about N = 4 to 8. This has the advantage that the total (or average) calculation can be performed several times (for the number of blocks) for one signal string while maintaining high accuracy in phase difference detection.

If the signal vector F of one or more types obtained by the CREW calculation unit 112 is Type 1 shown in FIG. 3 (b), Type 2 shown in FIG. 4 (b) is shown in FIG. 3 (c). If so, as shown in FIG. 4 (c), it becomes equivalent to the state where the frequency component of the wavelength N is mapped on the side of the square on the complex plane, and as shown in FIG. 6 (a), the power and It indicates the normalized phase.

In this way, the signal vector F for the captured image P1 obtained by the CREW calculation unit 112 is temporarily stored in the memory 12 and used as the reference signal vector F'for phase difference detection, and is obtained by the CREW calculation unit 112. The signal vector F for the captured image P2 is output to the phase difference calculation unit 113.

The phase difference calculation unit 113 reads out one or more types of reference signal vectors F'of the captured image P1 from the memory 12, and one or more types of signal vectors F and the one or more types of reference signal vectors F for the captured image P2 calculated this time. 'And are compared individually for each type, the phase difference φ is calculated so as to indicate the size change of the subject area, and is output to the coordinate calculation unit 114. As described above with reference to FIGS. 3 and 4, the phase difference calculation unit 113 calculates each signal vector mapped on the complex plane in calculating the phase difference φ of each signal vector obtained by the CREW calculation. Calculated from the Manhattan distance of. This contributes to the reduction of the processing cost related to the calculation of the phase difference φ.

The coordinate calculation unit 114 first reads out the coordinate calculation information indicating the wavelength λ and the displacement coordinate x from the memory 12, and is input from the image processing unit 112 based on the displacement coordinate x (initial value is zero). The region of interest of the captured images P1 and P2 of the above is calibrated. Subsequently, the coordinate calculation unit 114 multiplies the phase difference φ obtained from the phase difference calculation unit 113 and the wavelength λ in order to match the information of the phase difference φ with the position coordinates of the subject on the image captured image P2. The displacement coordinates (that is, the displacement coordinates of the region of interest) x of the subject on P2 are calculated, the information of the displacement coordinates x is output to the signal processing unit 13, and the memory 12 is updated and held.

The signal processing unit 13 corresponds to the phase difference φ of the subject reflected in both of the captured images P1 and P2 of the two images continuously acquired by the phase difference detector 11 in time. The subject area is cut out (section display) from the captured image P2 using the displacement coordinates x. After that, the signal processing unit 13 sequentially inputs the captured images acquired continuously in time by one imaging camera 10, and cuts out the subject region from the sequentially input captured images so as to surround the subject area, for example, in a square shape. It is repeated to display a section on the display unit 14 of a display panel or the like, or to record an image on the recording unit 15 in a manner indicating the cut out image area.

(Example of processing related to tracking a subject with one image pickup camera)
Hereinafter, a processing example of the image processing device 1 relating to the tracking of the subject by one image pickup camera 10 shown in FIG. 1 will be described more specifically.

FIG. 8 is a flowchart showing the processing of the image processing apparatus 1 including the phase difference detector 11 and the signal processing unit 13 of the first embodiment according to the present invention.

First, the phase difference detector 11 determines the wavelength λ by the main frequency detection unit 111 and holds it in the memory 12 based on the instruction from the operation unit 16 to start tracking the subject (start of the phase difference detection operation) (step S1). ).

Subsequently, the phase difference detector 11 divides the image Pn to be processed this time acquired by the image pickup camera by the CREW calculation unit 112 in one direction with a length of λ / 4, and performs a total calculation (or an average calculation) for each. (Step S2).

Subsequently, the phase difference detector 11 calculates the signal vector F of the frequency component of the image Pn by the CREW calculation unit 112 and holds it in the memory 12 (step S3).

Subsequently, the phase difference detector 11 reads the reference signal vector F'calculated by the phase difference calculation unit 113 from the memory 12 (it is set to zero if the reference signal vector F'is not held in the memory 12), and this time. The phase difference φ between the signal vector F of the calculated image Pn and the reference signal vector F'is calculated (step S4).

Here, when the phase difference calculation unit 113 is configured to obtain two or more types of signal vectors F and the two or more types of reference signal vectors F', the phase difference so as to indicate a change in the size of the subject area. φ can be calculated.

Subsequently, the phase difference detector 11 calculates the displacement coordinates x of the subject on the image Pn by φ * λ by the coordinate calculation unit 114, updates and holds them in the memory 12, and outputs them to the signal processing unit 13 (step S5). .. Here, the displacement coordinates x can include information indicating the size change of the subject area.

Subsequently, the signal processing unit 13 cuts out the subject area from the image Pn based on the displacement coordinates x and displays the section (step S6). Here, when the displacement coordinates x include the information indicating the size change of the subject region, the size of the region of interest can be updated with respect to the cutting out of the subject region. For example, even when the angle of view of a temporally continuous captured image changes due to zooming or the like, it is possible to cut out (partition display) the subject area in accordance with this.

After that, the phase difference detector 11 monitors whether or not there is an end instruction from the operation unit 16 (step S7), and when the operation unit 16 gives an end instruction (step S7: Y), the subject is tracked. (Operation of phase difference detection) is stopped, the signal processing unit 13 is notified to that effect, and the cutting (partition display) processing by the signal processing unit 13 is stopped.

On the other hand, the phase difference detector 11 continues tracking the subject (operation of phase difference detection) when there is no instruction to end from the operation unit 16 (step S7: N), and if necessary in design, to that effect. Notify the signal processing unit 13.

When continuing the tracking of the subject (operation of phase difference detection), the phase difference detector 11 may shift to step S2, but here, as a more preferable example, it shifts to step S8.

Specifically, when the tracking of the subject (operation of phase difference detection) is continued, the phase difference detector 11 renews the said in the image Pn based on the information of the displacement coordinates x related to the cutout by the signal processing unit 13. The cut-out subject area is divided by the CREW calculation unit 112 in one direction with a length of λ / 4, and a total calculation (or an average calculation) is performed for each (step S8).

Subsequently, the phase difference detector 11 recalculates the signal vector F (here, referred to as a signal vector Fc for distinction) by the CREW calculation unit 112 for the cut-out subject area in the image Pn (step). S9).

Subsequently, the phase difference detector 11 compares the signal vector Fc with the calculated signal vector F by the CREW calculation unit 112 (step S10), and the difference between the signal vector Fc and the signal vector F is equal to or less than the threshold value. Whether or not it is determined (step S11). If the difference between the signal vector Fc and the signal vector F is not equal to or less than the threshold value (step S11: N), the process proceeds to step S1, and if the difference between the signal vector Fc and the signal vector F is not equal to or less than the threshold value (step S11: Y). ), The process proceeds to step S12.

When the process proceeds to step S12, the phase difference detector 11 uses the signal vector Fc as the reference signal vector F'as the memory 12 for analysis of the newly input image Pn acquired by the image pickup camera 10 by the CREW calculation unit 112. The update is held in and the process proceeds to step S2.

That is, when the CREW calculation unit 112 continuously detects the phase difference for images that are continuous in the time direction, the signal vector Fc of the image to be calculated this time is sequentially calculated in the subject area cut out this time. It is recalculated as a value and compared with the signal vector F of the previous value for the image to be calculated this time calculated in the subject area cut out last time, and the difference between the signal vector Fc of the current value and the signal vector F of the previous value is the threshold value. It is determined whether or not it is as follows. Then, the CREW calculation unit 112 updates the signal vector Fc of the current value as the reference signal vector F'to be compared in the next detection of the phase difference when it is below the threshold value, and when it is not below the threshold value, the reference signal vector F'is updated. It is possible to configure the main frequency detection unit 111 to have a process of calculating the wavelength λ again without updating.

In this way, by providing the CREW calculation unit 112 with the processes of steps S9 to S12, it is robust even when the subject suddenly decreases between a set of images in the time direction for which the phase difference is to be detected. The reference signal vector F'and the subject area can be updated.

In the description of the present embodiment, an example has been described in which one attention area of a subject is initially set and one subject area of the subject is cut out for tracking, but the attention areas of a plurality of subjects are initially set. Then, it can be configured to cut out a plurality of subject areas individually. In this case, an identifier is assigned to the region of interest of each subject that is updated by the subject region that is cut out between the captured images that are continuous in time, and the phase difference detection target is identified based on the identifier. good.

In this way, the image processing device 1 can be configured to perform tracking processing of the subject by one image pickup camera 10 shown in FIG. In particular, in the case of a configuration in which a plurality of types of signal vectors are used for one image signal, the displacement coordinates x can include information indicating the size change of the subject area. Therefore, regarding the cutting out of the subject area, the size of the area of interest can be updated. For example, as shown in FIG. 9, in the captured image Pn (t1) at time t1, when the subject area displayed as a cutout section for the subject Obj indicated by the broken line is SQ (t1), the time t2 that follows in time. In the captured image Pn (t2), it is also possible to track the subject Obj shown by the solid line while changing the size of the subject area to be cut out by the subject area displayed in the cutout section as in SQ (t2).

By applying the process shown in FIG. 8, it is possible to detect the moving speed of the subject on the image from the moving amount of the subject and the number of frames.

As described above, according to the phase difference detector 11 of the present embodiment, the phase difference can be detected stably and with high accuracy at a low processing cost regardless of the photographing environment.

FIG. 10 is a diagram showing the operation of the image processing device 1 provided with the phase difference detector 11 of the first embodiment according to the present invention and its action / effect. As shown in FIG. 10, the compressed image summed (or averaged) in one direction in the processing process by the CREW calculation unit 112 with respect to the original image captured by the subject with the background is "no displacement" of the original image. The height of the luminance distribution changes roughly according to "right displacement (phase advance)" and "left displacement (phase return)" (stable regardless of the shooting environment). When the CREW filter of Type 2 is applied to this compressed image by the CREW calculation unit 112, for example, even when the amplitude b = 1 of the CREW filter, the signal vector on the complex plane is "displaced" of the original image as shown in the figure. The phase changes significantly according to "none", "right displacement (phase advances)", and "left displacement (phase returns)". Then, the phase difference according to "no displacement", "right displacement (phase advances)", and "left displacement (phase returns)" of the original image can be detected at low processing cost and with high accuracy.

According to the present invention, since the signal vector indicating the power and the normalized phase is used as the evaluation value, the number of significant digits of the shift width k of the phase difference image by the evaluation value based on the equation (Equation 1) as in the conventional technique. It can be an evaluation value with a higher number of significant digits. As a result, according to the phase difference detector 11 of the present embodiment, the phase difference detection can be made highly accurate and the mounting cost can be suppressed to a low level. Further, the phase difference detector 11 of the present embodiment is not limited to the use of detecting the movement speed of the subject on the image and tracking the movement of the subject, but also detects the dynamic change of the subject (dynamic strain inspection of the object, etc.). ), It can be applied to any application that can utilize the detection of the phase difference indicating the deviation of the position of the subject from a set of images in the time direction.

[Second Embodiment]
FIG. 11 is a block diagram showing a schematic configuration of a phase difference detector 11 according to a second embodiment of the present invention and an image processing device 1 including the same. The same reference numbers are assigned to the same components as those in the first embodiment. The image processing device 1 of the present embodiment is a set of two images acquired spatially simultaneously and in the same direction by a binocular camera 10RL in which two image pickup cameras 10R and 10L are spatially arranged and imaged simultaneously and in the same direction. By inputting the captured image of the above and detecting the phase difference of the subject reflected on both of them based on the input image of a pair of images in the spatial direction by the phase difference detector 11, the image pickup camera is concerned. It is configured as a device for detecting the distance between 10R and 10L and the subject, or the spatial position between the subjects.

(Binocular camera)
First, each of the image pickup cameras 10R and 10L constituting the binocular camera 10RL according to the present embodiment includes an image pickup lens 101, an image pickup element 102, and an image processing unit 103 having the same performance, respectively.

The image pickup lens 101 according to the present embodiment is an optical lens for forming an image of a subject on the image pickup element 102, and may be driven so as to be able to automatically focus. However, here, a state in which the focus is on a desired subject is set. Let's take an example.

The image pickup device 102 according to the present embodiment has an image pickup unit composed of a plurality of pixels, and has phase difference detection pixels having different light-shielding portions (generally symmetrical in the left-right and up-down directions) in the image pickup unit. However, here, an example of not having such a phase difference detection pixel will be described.

The image processing unit 103 according to the present embodiment sequentially forms an image captured by a subject obtained from the image sensor 102 in frame units, and outputs the image to the signal processing unit 13 and the phase difference detector 11. In particular, the image processing unit 103 in each of the image pickup cameras 10R and 10L forms the image pickup images Pr and Pl in which the subject obtained from each image pickup element 102 is imaged at the time when the operation of the image phase difference detector 11 starts, and signals. It is output to the processing unit 13 and the phase difference detector 11.

(Image processing device)
Similar to the first embodiment, the image processing device 1 includes a phase difference detector 11, a memory 12, and a signal processing unit 13, and the phase difference detector 11 captures a set of two images in the spatial direction instead of the time direction. The operation is almost the same as that of the first embodiment except that the image is handled.

The phase difference detector 11 initially sets one or more areas of interest (regardless of vertical, horizontal, or diagonal) of the subject based on an instruction from the operation unit 16 and starts operation, and spatially with the binocular camera 10RL. It is a functional unit that inputs a set of two images acquired at the same time and in the same direction and detects the phase difference of the subject reflected on both of them. The initial setting of the area of interest of the subject can be specified by the operator at an arbitrary position, or can be set to a predetermined range such as the central area of the captured image. At the start of operation of the image phase difference detector 11, the initially set area of interest of the subject is dynamically calculated in coordinates by the subsequent operation of the image phase difference detector 11 so as to surround the subject in a square shape, for example. It is cut out.

The memory 12 has a function of temporarily storing captured images Pr and Pl required for processing of the signal processing unit 13 while sequentially updating them, and a function of temporarily storing each signal value required for processing of the phase difference detector 11. .. In the memory 12 of this example, the arrangement information of the imaging cameras 10R and 10L constituting the binocular camera 10RL is preset and stored via the operation unit 16.

The signal processing unit 13 measures the phase difference φ of the subject reflected in both of them, which is detected based on the pair of captured images Pr and Pl spatially acquired simultaneously and in the same direction by the phase difference detector 11. It is a functional unit that cuts out a subject area from each of the captured images Pr and Pl using the corresponding displacement coordinates x and detects the distance between the imaging cameras 10R and 10L and the subject or the spatial position between the subjects by calculation. Therefore, the signal processing unit 13 sequentially inputs a set of two captured images Pr and Pl that are spatially acquired simultaneously and in the same direction by the binocular camera 10RL, and sequentially updates the captured images Pr and Pl. Temporarily stored in the memory 12. Then, for example, the signal processing unit 13 cuts out the subject area from the sequentially input captured images so as to surround the subject region, for example, and determines the distance between the imaging cameras 10R and 10L and the subject, or the spatial position between the subjects. Repeat the detection. The distance from the binocular camera 10RL to each subject is calculated by the arrangement information of each image pickup camera 10R, 10L including the distance between the image pickup cameras 10R, 10L of the binocular camera 10RL and the angle of view, and the phase difference detector 11. It can be obtained by calculation based on the phase difference φ (difference in the normalized phase θ ′ of the signal vector to be compared).

More specifically, the phase difference detector 11 and the image processing device 1 will be described in detail below.

(Phase difference detector)
As shown in FIG. 11, the phase difference detector 11 of the present embodiment includes a main frequency detection unit 111, a CREW calculation unit 112, a phase difference calculation unit 113, and a coordinate calculation unit 114, as in the first embodiment. ..

The main frequency detection unit 111 operates so as to calculate the wavelength λ of the main frequency of either one or both of the captured images Pr and Pl at the time when the operation of the image phase difference detector 11 starts. That is, the main frequency detection unit 111 exemplifies the region of interest of the captured images Pr and Pl input from the image processing unit 103 at the start of operation of the image phase difference detector 11 with reference to FIG. 2 described above. The detection is performed, the wavelength λ of the main frequency at which the power (amplitude of the pixel value) of the captured images Pr and Pl is maximized is calculated, and the wavelength λ is temporarily stored in the memory 12. Either one of the captured images Pr and Pl may be calculated, but when calculating the wavelength λ of both main frequencies, the average value is used.

The CREW calculation unit 112 first reads out the coordinate calculation information indicating the arrangement information of the image pickup cameras 10R and 10L constituting the binocular camera 10RL and the wavelength λ, the number of divided blocks N, from the memory 12, and the respective image pickup cameras 10R and 10L. The attention area of the two images Pr and Pl input from the image processing unit 112 is calibrated based on the arrangement information of the above, and the attention area of the two images Pr and Pl based on the wavelength λ. By dividing each of these into N blocks or more (fixed integer value of N ≧ 4) in one direction at a wavelength of λ / 4 and calculating the sum (or average) of each, a compressed image with N pixels in the dividing direction can be obtained. Generate.

For example, by initializing one or more areas of interest of the subject based on the instruction from the operation unit 16 for the image captured image Pr from the image pickup camera 10R, each image pickup camera is set from the image pickup camera 10L to the image capture image Pl. According to the arrangement information of 10R and 10L, the setting of the region of interest of the corresponding subject can be calibrated. It should be noted that the areas of interest of each of the imaging cameras 10R and 10L do not need to be strictly matched with respect to the subject.

Then, the CREW calculation unit 112 has element values of at least four quadrants in the direction of the compressed image having N pixels in the division / compression direction in each of the attention regions of the captured images Pr and Pl of the two sheets. A complex rectangular wavelet transform (CREW) based on a complex rectangular sequence (CREW filter) represented by a rectangular sequence is applied, and the complex (real and imaginary component) frequency components are used for the right-eye image and the left-eye image. One or more types of signal vectors Fr and Fl are generated and output to the phase difference calculation unit 113.

The processing itself of the CREW calculation unit 112 in this embodiment is the same as that of the first embodiment described above, and can be a CREW filter of either Type 1 or 2.

The image to be processed by the CREW calculation unit 112 is based on the attention area of the subject initially set based on the instruction from the operation unit 16 in this example in order to further reduce the processing cost. It may be the whole.

The phase difference calculation unit 113 individually compares one or more types of signal vectors Fr and Fl for each of the captured images Pr and Pl calculated this time, calculates the phase difference φ for each region of interest of one or more subjects, and calculates the coordinates. It is output to the calculation unit 114. As described above with reference to FIGS. 3 and 4, the phase difference calculation unit 113 calculates each signal vector mapped on the complex plane in calculating the phase difference φ of each signal vector obtained by the CREW filter. Calculated from the Manhattan distance of. This contributes to the reduction of the processing cost related to the calculation of the phase difference φ.

The coordinate calculation unit 114 first reads out the coordinate calculation information indicating the arrangement information of each image pickup camera 10R, 10L constituting the wavelength λ and the binocular camera 10RL from the memory 12, and if necessary for designing, each image pickup camera 10R, Based on the arrangement information of 10L, the region of interest of the pair of captured images Pr and Pl input from the image processing unit 112 is calibrated. Subsequently, the coordinate calculation unit 114 pays attention to one or more subjects obtained from the phase difference calculation unit 113 in order to restore the information of the phase difference φ to the displacement amount on the coordinates of each of the focused areas of the captured images Pr and Pl. The displacement coordinates (that is, the displacement coordinates of the region of interest) x of the subject on the captured images Pr and Pl are calculated by multiplying the phase difference φ and the wavelength λ for each region, and the information of the displacement coordinates x is output to the signal processing unit 13. At the same time, it is updated and held in the memory 12.

The signal processing unit 13 is a region of interest of one or more subjects reflected in both of the captured images Pr and Pl detected by the phase difference detector 11 based on a pair of captured images Pr and Pl spatially acquired simultaneously and in the same direction. The subject area is cut out from each of the captured images Pr and Pl (section display) using the displacement coordinates x corresponding to the phase difference φ for each, and the distance between the imaging cameras 10R and 10L and the subject, or the spatial position between the subjects. Is detected by calculation. The signal processing unit 13 reads out the arrangement information of each image pickup camera 10R, 10L from the memory 12 when detecting the distance between the image pickup cameras 10R, 10L and the subject or the spatial position between the subjects by calculation. (Not shown in FIG. 11).

After that, the signal processing unit 13 sequentially inputs each of the captured images spatially acquired simultaneously and in the same direction by the respective imaging cameras 10R and 10L constituting the binocular camera 10RL, and sequentially inputs the subject area from the sequentially input captured images. For example, it is cut out so as to surround it in a square shape, information indicating the above distance and spatial position is given, and the image is divided and displayed on the display unit 14 of a display panel or the like, or recorded in a mode indicating the cut out image area. The image recording for the unit 15 is repeated.

(Processing example related to distance measurement with a binocular camera)
Hereinafter, a processing example of the image processing device 1 relating to the distance measurement between the image pickup cameras 10R and 10L and the subject by the binocular camera 10RL shown in FIG. 11 will be described more specifically.

FIG. 12 is a flowchart showing the processing of the image processing apparatus 1 including the phase difference detector 11 and the signal processing unit 13 of the second embodiment according to the present invention.

First, the phase difference detector 11 determines the wavelength λ by the main frequency detection unit 111 and holds it in the memory 12 based on the instruction from the operation unit 16 to start the distance measurement to the subject (start the operation of the phase difference detection). Step S21).

Subsequently, the phase difference detector 11 divides each region of interest of the captured images Pr and Pl to be processed this time acquired by the binocular camera 10RL by the CREW calculation unit 112 in one direction with a length of λ / 4. A total calculation (or an average calculation) is performed for each (step S22).

Subsequently, the phase difference detector 11 calculates the signal vectors Fr and Fl of the frequency components of the respective attention regions of the captured images Pr and Pl by the CREW calculation unit 112 and holds them in the memory 12 (step S23).

Subsequently, the phase difference detector 11 calculates the phase difference φ between the signal vectors Fr and Fl of the frequency components for each region of interest of the captured images Pr and Pl by the phase difference calculation unit 113 (step S24).

Here, when the phase difference calculation unit 113 is configured to obtain two or more types of signal vector Fr and the two or more types of reference signal vector Fl as in the first embodiment, the size of the subject area. The phase difference φ can be calculated to indicate the change.

Subsequently, the phase difference detector 11 calculates the displacement coordinates x of the subject on the image Pn based on the arrangement information of the imaging cameras 10R and 10L in the binocular camera 10RL and φ * λ by the coordinate calculation unit 114. Then, it is output to the signal processing unit 13 (step S25). Here, the displacement coordinates x can include information indicating the size change of the subject area.

Subsequently, the signal processing unit 13 cuts out a subject area from each of the captured images Pr and Pl using the displacement coordinates x, and calculates the distance between the imaging cameras 10R and 10L and the subject or the spatial position between the subjects. Detect (step S26). Here, when the displacement coordinates x include the information indicating the size change of the subject region, the size of the region of interest can be updated with respect to the cutting out of the subject region. For example, even when the angle of view of a temporally continuous captured image changes due to zooming or the like, it is possible to cut out (partition display) the subject area in accordance with this.

After that, the phase difference detector 11 monitors whether or not there is an end instruction from the operation unit 16 (step S27), and when there is an end instruction from the operation unit 16 (step S27: Y), the subject and the subject. The distance measurement (operation of phase difference detection) is stopped, the signal processing unit 13 is notified to that effect, and the cutting (partition display) processing by the signal processing unit 13 is stopped.

On the other hand, when the operation unit 16 does not instruct the end of the phase difference detector 11 (step S27: N), the phase difference detector 11 continues to measure the distance to the subject (operation of the phase difference detection), and if necessary for designing, the phase difference detector 11 continues to measure the distance to the subject. Notify the signal processing unit 13 to that effect.

When continuing the distance measurement with the subject (operation of phase difference detection), the phase difference detector 11 may move to step S2 and continue to use the wavelength λ held in the memory 12 during the continuation. In this example, the process proceeds to step S1 in order to adaptably respond to changes in the angle of view such as zooming.

In this way, the image processing device 1 can be configured to perform distance measurement processing with the subject by the binocular camera 10RL shown in FIG. In particular, in the case of a configuration in which a plurality of types of signal vectors are used for one image signal, the displacement coordinates x can include information indicating the size change of the subject area. Therefore, regarding the cutting out of the subject area, the size of the area of interest can be updated. In addition, distance measurement can be performed on a plurality of subjects in the captured image. For example, as shown in FIG. 13, in the captured image Pr of the imaging camera 10R (or the captured image Pl of the imaging camera 10L), the subject areas SQ1, SQ2, SQ3 are individually cut out and displayed for a plurality of subjects Obj1, Obj2, Obj3. Distance measurement can be performed by setting each of the above to the distance measuring areas 1, 2, and 3, respectively. Therefore, as one of the physical distance measurement processes relating to the subject, the distance between the imaging cameras 10R and 10L constituting the binocular camera 10RL and the subject, or the spatial position between the subjects is detected. be able to.

Further, in the second embodiment, instead of the binocular camera 10RL, an image pickup lens is used as a drive-controllable image pickup camera, and the light-shielding portion is different in the pixel portion of the image pickup element (generally, it is symmetrical in the left-right and up-down directions). ) By providing the phase difference detection pixel, as one of the physical distance measurement processes relating to the subject, the phase difference detector 11 detects the phase difference for automatic focus adjustment by the same process. Can be configured as.

Since the phase difference detector 11 according to the second embodiment has substantially the same operation as that of the first embodiment, it is stable and highly accurate at low processing cost regardless of the photographing environment as in the first embodiment. The phase difference can be detected with.

Although the present invention has been described above with reference to examples of specific embodiments, the present invention is not limited to the above-mentioned examples, and can be variously modified without departing from the technical idea. For example, in each of the above-described embodiments, the CREW calculation unit 112 has described a one-dimensional CREW filter that is spatially targeted in one direction as an example, but a two-dimensional CREW filter that is spatially targeted in two directions. , And a CREW filter having three or more dimensions including the time direction in addition to the two directions.

Further, in each of the above-described embodiments, a preferred example in which the CREW calculation unit 112 performs the CREW calculation after generating a compressed image in order to stabilize the processing cost and operation has been described. The CREW filter is made redundant by continuously connecting each element value of the above complex rectangular wavelet (CREW filter) by λ / 4 directly to the uncompressed processed target image without generating the compressed image of. By performing the CREW calculation, it is possible to obtain a signal vector equivalent to that in the case of generating a compressed image. In this case, for example, in a system (on hardware) where pixel values can be obtained serially, the pixel values of the uncompressed image can be sequentially added or subtracted without buffering.

Further, in each of the above-described embodiments, the specific use has been described in detail, but the phase difference detector 11 according to the present invention has a phase difference indicating a deviation of the position of the subject from a set of images in the time direction or the space direction. Can be used for various purposes as long as it can be used.

According to the present invention, since the phase difference can be detected stably and with high accuracy regardless of the shooting environment at a low processing cost, the phase difference indicating the deviation of the position of the subject from a set of images in the time direction or the space direction. It is useful for applications that make it available.

1 Image processing device 10 Imaging camera 10RL Binocular camera 10R Right eye imaging camera 10L Left eye imaging camera 11 Phase difference detector 12 Memory 13 Signal processing unit 14 Display unit 15 Recording unit 16 Operation unit 101 Imaging lens 102 Imaging element 103 Image processing unit 111 Main frequency detection unit 112 Complex rectangular wavelet transform (CREW) calculation unit 113 Phase difference detection unit 114 Coordinate calculation unit

Claims (13)

A phase difference detector that detects a phase difference indicating a shift in the position of a subject from a set of images in the temporal or spatial direction.
A main frequency detecting means for detecting the wavelength of the main frequency of the image signal in each region of interest of the set of images,
The image signal of each attention region of the set of images is used as the processing target image, and the complex rectangular wavelet transform (CREW) is applied to the processing target image based on the frequency, and each attention region of the set of images is applied. A CREW calculation means that generates a signal vector of complex frequency components related to the image signal of
A phase difference calculation means for calculating the phase difference by comparing the signal vectors of each region of interest of the set of images, and
Based on the calculated phase difference, a coordinate calculation means for calculating the displacement coordinates of the subject in at least one of the attention regions of the set of images, and a coordinate calculation means.
A phase difference detector characterized by comprising. The phase difference detector according to claim 1, wherein the basis of the complex rectangular wavelet transform is represented by a filter of a complex rectangular sequence having element values of at least four quadrants. The phase difference detector according to claim 1 or 2, wherein the phase difference calculating means calculates the phase difference by the Manhattan distance of each signal vector mapped on a complex plane. The CREW calculation means spatially divides and compresses the image to be processed in at least one direction based on the wavelength, and generates a compressed image in which the number of pixels in the one direction is N (N ≧ 4). The phase difference detector according to any one of claims 1 to 3, wherein a complex rectangular wavelet transform (CREW) is performed on the compressed image. The CREW calculation means is a complex rectangular wavelet transform based on a CREW filter having element values of four quadrants for a compressed image having N (N ≧ 4) pixels in one direction divided and compressed at 1/4 of the wavelength. 4. The phase difference detector according to claim 4. The CREW calculation means is a complex rectangular wavelet transform based on a CREW filter having element values of four quadrants for a compressed image having N (N> 4) pixels in one direction divided and compressed at 1/4 of the wavelength. The phase difference detector according to claim 4 or 5, wherein a plurality of types of signal vectors are generated according to the address of the pixel. The CREW calculation means directly performs a CREW calculation on the image to be processed by making each element value of the complex rectangular wavelet transform continuous by 1/4 of the wavelength to make the CREW filter redundant. The phase difference detector according to any one of claims 1 to 3, wherein the phase difference detector is characterized. The CREW calculation means, as the complex rectangular wavelet transform, is a one-dimensional CREW filter that is spatially targeted in one direction, a two-dimensional CREW filter that is spatially targeted in two directions, or a time added to the two directions. The phase difference detector according to any one of claims 1 to 7, wherein the wavelet transform is performed based on a CREW filter having three or more dimensions including a direction. The phase difference detector is configured to sequentially detect the phase difference from continuous images in the time direction.
The main frequency detecting means monitors the means for calculating the wavelength as an initial value when sequentially detecting the phase difference and the power of the signal vector generated by the CREW calculating means, and the power of the signal vector is used. The phase difference detector according to any one of claims 1 to 8, further comprising means for recalculating the wavelength when the wavelength deviates from a predetermined range. The phase difference detector is configured to sequentially detect the phase difference from continuous images in the time direction.
When the CREW calculation means continuously detects the phase difference for continuous images in the time direction, the signal vector is sequentially recalculated as the current value and compared with the signal vector of the previous value, and this time. It is determined whether or not the difference between the value signal vector and the previous value signal vector is less than or equal to the threshold value, and if it is less than or equal to the threshold value, the current value signal vector is updated as a reference signal vector to be compared in the next phase difference detection. However, if it is not equal to or less than the threshold value, the reference signal vector is not updated, and the main frequency detecting means has a process of recalculating the wavelength, which is any one of claims 1 to 9. The phase difference detector described in the section. The phase difference detector according to any one of claims 1 to 10, which is configured to sequentially detect the phase difference from images continuous in the time direction.
A signal processing means for detecting the moving speed of the subject on the image, tracking the movement of the subject, or detecting the dynamic change of the subject based on the phase difference.
An image processing device characterized by comprising. The phase difference detector according to any one of claims 1 to 10, which is configured to sequentially detect the phase difference from a set of images simultaneously and simultaneously captured in the spatial direction.
A signal processing means that performs physical distance measurement processing related to the subject based on the phase difference, and
An image processing device characterized by comprising. A program for causing a computer to function as the phase difference detector according to any one of claims 1 to 10.

Images