JP4147656B2 - Image processing device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing apparatus that generates a stereoscopic image, and more particularly to an image processing apparatus characterized by correction processing for tilt and position of two images.
[0002]
[Prior art]
With the rapid spread of computer systems, particularly personal computers, in recent years, the demand for various peripheral devices is also increasing. In particular, as a peripheral device for capturing an image in a personal computer, a digital still camera that can directly capture a captured photographic image as digital data has attracted attention.
[0003]
In general, an image captured by a digital still camera is printed out after being subjected to some image processing.
The image processing apparatus proposed in Japanese Utility Model Publication No. 9-416 detects the overlapping position of images adjacent to each other when given a plurality of images that are captured in such a manner that their edges overlap each other. In such a situation, a large subject that does not fit within the imaging range of one image is divided into a plurality of partial areas, for example, by printing them together as a single image at that position, and imaging over a plurality of images The purpose is to obtain a large print output with good resolution.
[0004]
By the way, when two images taken at a certain interval are juxtaposed with respect to the same subject, and each image is associated with either the right eye or the left eye and directly viewed, the person becomes the subject in the image. It is generally known to feel depth. In the following, a set of two images for providing such an effect is referred to as a stereo image. This stereo vision image is often used for learning materials and toys.
As an apparatus for creating a stereoscopic image, two objective lenses respectively associated with the right eye and the left eye are arranged at a predetermined interval, and the light beam transmitted through each objective lens is reflected by a mirror or a prism. There are cameras that are configured such that they are imaged in the right and left halves of the screen, respectively.
[0005]
In addition, based on the same principle as this stereo image, two systems of moving images captured by two video cameras installed at regular intervals are alternately displayed on the television screen at a predetermined timing. In addition, in synchronization with this display switching timing, a video reproduction device that gives viewers a sense of depth by alternately blocking the viewer's right eye or left eye field of view with goggles worn by the viewer is already a product It has become.
[0006]
[Problems to be solved by the invention]
In such a stereoscopic image, the effect obtained when there is a gap between the imaging ranges of the respective images associated with the right eye and the left eye is diminished. It is necessary to pay close attention as much as possible. In particular, when two imaging systems are used as in the above-described moving image playback apparatus and images associated with the right eye and the left eye are separately captured, the center position of each other's imaging range from the start to the end of imaging In addition, since it is necessary to maintain a state in which the inclinations coincide with each other, handling is not easy. Below, it demonstrates using drawing.
[0007]
FIG. 8 is an explanatory diagram for explaining a shift between the imaging ranges of the two imaging systems. C _R and C _L in the figure, an imaging system that is associated with the respective right and left eyes, and represents the imaging range in the rectangular area R and L. In addition, the region X where the rectangular regions R and L overlap is indicated by hatching.
The region X is a region in which an expected effect can be obtained for a stereo image. However, the greater the displacement between the rectangular regions R and L, the less effective the effect.
[0008]
In the above-described camera that captures a stereo image, the right eye and the left eye are associated with each other with two objective lenses arranged at a predetermined interval, and a mechanism for guiding the light transmitted through the objective lens. There was a problem that it became an obstacle to downsizing.
[0009]
The present invention has been made in view of such circumstances, and accepts input of two images or moving images individually associated with the right eye and the left eye, respectively, and corrects the shift of the imaging range between the systems. Thus, an object of the present invention is to provide an image processing apparatus that facilitates handling of the camera at the time of imaging and is advantageous for downsizing the apparatus.
[0010]
[Means for Solving the Problems]
An image processing apparatus according to a first aspect of the present invention is an image processing apparatus that performs image processing for matching the inclinations of both subjects with respect to two images obtained by capturing the same subject, performing orthogonal transformation on each image, Means for obtaining each conversion coefficient, means for detecting the peak positions of the luminance levels on a plurality of scanning lines in a predetermined direction, and a plurality of detected intensity distribution images for the obtained intensity distribution images of the respective conversion coefficients Means for determining an angle formed by line segments connecting pixels at the peak position, and means for correcting the inclination of one image so as to reduce the angle based on the determined angle.
[0011]
FIG. 6 is an explanatory diagram for explaining the concept of the present invention. In the figure, A represents an example of an image, and P represents a power spectrum (intensity distribution) image of a frequency component obtained by subjecting the image A to a two-dimensional discrete Fourier transform (hereinafter referred to as DFT). .
The power spectrum image P is an image displayed in gradation based on the intensity of the frequency component using a polar coordinate system with the center of the image as the origin. Here, in the power spectrum image P, only its characteristic partial region is schematically represented by a line segment U having a substantially rhombus shape, and the intensity expression of the luminance level of each pixel is omitted. Specifically, the line segment U is a line segment that connects pixels with substantially the same luminance level, and in the actual power spectrum image P, the inner side of the line segment U becomes closer to the origin, and the luminance level becomes higher. In addition, the brightness level decreases as the distance from the origin increases.
[0012]
On the other hand, B represents an example of another image captured with respect to the same subject as the image A, and the imaging range and the imaging range of the image A have a slight deviation in inclination. Q represents a power spectrum image of a frequency component obtained by applying DFT to the image B, and the density expression is based on the power spectrum image P.
[0013]
When the power spectrum of the DFT is displayed on the polar coordinate system in this way, it is generally known that a characteristic related to the directionality of the image appears as a change in luminance level in the vicinity of the angle corresponding to the directionality of the power spectrum image. ing.
For example, an image obtained by adjusting an outdoor landscape so that its horizontal line is parallel to the horizontal direction of the imaging range has a general characteristic that the horizontal component and the vertical component increase in directionality. In this case, in the power spectrum image of the DFT of the image, a luminance level peak appears along the horizontal and vertical line segments passing through the origin.
[0014]
In other words, when the peak of the luminance level appears along the line segment in the direction different from the horizontal direction and the vertical direction in the power spectrum image of the DFT for the image captured of the outdoor scenery, the imaging range adjusted at the time of imaging It can be discriminated that the horizontal direction is inclined from the horizontal line of the outdoor scenery.
As for the inclination, the line segment connecting the pixels at the peak position when the power spectrum image is scanned in the vertical direction (corresponding to the horizontal component of the captured image) or the pixel at the peak position when scanning in the horizontal direction is connected. The line segment (corresponding to the vertical component of the captured image) can be obtained from the angle formed with the horizontal direction or the vertical direction of the power spectrum image, respectively.
[0015]
The line segment D _X and the line segment D _Y represented by broken lines in the power spectrum image Q described above are line segments connecting pixels at the peak positions when the power spectrum image P of the image A is scanned in the vertical direction and the horizontal direction, respectively. The angle formed by this and an equivalent line segment for image B is obtained and used for inclination correction.
[0016]
Specifically, the power spectrum image Q is scanned in the vertical direction (arrow Q _Y ) and the horizontal direction (arrow Q _X ), respectively, and the peak position of the brightness level is detected individually, and the pixel at the detected peak position is detected. The connected line segment d _X and line segment d _Y are obtained.
Next, an angle formed by the obtained line segment d _X and line segment d _Y and the corresponding line segment D _X and line segment D _Y is obtained. Since both not always agree, an average between the angle and the line segment d _Y and the line segment D _Y is the angle which the line segment d _X and the line segment D _X forms, this constant by the angle of the object Ensure accuracy.
Then, the inclination of the image B is corrected in the direction in which the angle becomes smaller, so that the inclination of the image B matches the inclination of the image A.
[0017]
In the present invention, since the deviation of the inclination of the imaging range between the systems is corrected by image processing, the camera can be easily handled during imaging. Further, since the images or moving images of each system received individually are processed, it is not necessary to provide a mechanism for simultaneously forming images associated with the right eye and the left eye on one screen.
[0018]
An image processing apparatus according to a second aspect of the present invention is an image processing apparatus that performs image processing for matching the inclinations of both subjects with respect to two images obtained by imaging the same subject, and performs orthogonal transformation on each image, Means for obtaining each conversion coefficient, means for detecting the peak positions of the luminance levels on a plurality of scanning lines in a predetermined direction, and a plurality of detected intensity distribution images for the obtained intensity distribution images of the respective conversion coefficients Means for determining the angle formed by the line segments connecting the pixels at the peak position, means for correcting the inclination of one intensity distribution image so as to reduce the angle based on the calculated angle, and the corrected intensity distribution image Means for performing orthogonal inverse transform on the transform coefficient to obtain an image.
[0019]
According to the present invention, the inclination of the power spectrum image Q is corrected in the direction in which the angle is reduced based on the obtained angle, and the two-dimensional discrete Fourier inverse transform (hereinafter referred to as IDFT) is performed on the corrected image. An image B in which the inclination matches the inclination of the image A can be obtained.
[0020]
In the present invention, since the deviation of the inclination of the imaging range between the systems is corrected by image processing, the camera can be easily handled during imaging. Further, since the images or moving images of each system received individually are processed, it is not necessary to provide a mechanism for simultaneously forming images associated with the right eye and the left eye on one screen.
[0021]
An image processing apparatus according to a third aspect of the present invention is an image processing for performing image processing for matching the positions of both subjects with respect to two images obtained by capturing the same subject in order to generate a stereoscopic image. A device that cuts out a region of a predetermined size at a predetermined position of one image, a device that obtains a first inspection image by applying a high-pass filter to the cut-out region of the image, At the predetermined position and a plurality of different positions and a predetermined number of previously input positions , a region having the same size as the clipped region is cut out, and a high-pass filter is applied to each cut out region of the image. And, for each set of the obtained first inspection image and each of the second inspection images, the sum of the luminance level differences between the pixels corresponding to the positions is calculated. Means and performance Means for comparing each sum and identifying a set having a small size, and means for correcting the position of one image so as to reduce this difference based on the difference in the cutout position of each area in the specified set. It is characterized by providing.
[0022]
According to a fourth aspect of the present invention, in the image processing apparatus according to the second aspect, the high-pass filter is configured to obtain a transform coefficient by performing orthogonal transform on a region where the image is cut out, and a low-frequency component of the obtained transform coefficient. And means for performing orthogonal inverse transform on the transform coefficient from which the low-frequency component has been removed.
[0023]
FIG. 7 is an explanatory diagram for explaining another concept of the present invention. In the figure, E represents an example of an image, and R represents a filter image obtained by applying a high-pass filter to a partial region of the image E. Here, the filter image R schematically represents only the features by a straight line in the vertical direction, and the gradation expression of the luminance level of each pixel is omitted. Specifically, the luminance level on the left side and the luminance level on the right side of the partial area greatly change with the approximate center as a boundary. In other words, the straight line in the vertical direction in the filter image R corresponds to the edge portion between the hatched area of the image E and another area.
[0024]
On the other hand, F represents an example of another image captured with respect to the same subject as the image E, and the imaging range and the imaging range of the image E have a slight shift in position. S represents a filter image obtained by applying a high-pass filter to the image F, and the gradation expression is based on the filter image R.
In the filter image R and the filter image S obtained in this way, a difference in luminance level is calculated for each of the two pixels, and a position where the sum of the differences becomes the smallest is obtained and used for the correction.
[0025]
Specifically, a rectangular area of a predetermined size is first cut out near the center of the image E, and a high-pass filter is applied to this to obtain a filter image R. Next, the rectangular area is cut out near the center of the image F, and a high-pass filter is applied to the rectangular area to obtain a filter image S. Further, the difference in luminance level is calculated between the pixels corresponding to the positions of the filter image R and the filter image S, and the sum is calculated.
[0026]
Furthermore, the rectangular region is cut out at a position slightly moved in either direction from the position related to the previous cut-out of the image F, and a high-pass filter is applied to the rectangular region to obtain a new filter image S. The sum of differences in luminance levels between pixels corresponding to the positions of R and filter image S is calculated.
Then, the position where the calculated sum is the smallest is specified, and the positional deviation of the position is corrected with respect to the entire image based on the relative positional relationship between the position and the position of the filter image R. A matched image can be obtained.
[0027]
In the third invention as claimed in claim 1 and the fourth invention as claimed in claim 2, since the shift of the position of the imaging range between systems is corrected by image processing, handling of the camera at the time of imaging becomes easy. Further, since the images or moving images of each system received individually are processed, it is not necessary to provide a mechanism for simultaneously forming images associated with the right eye and the left eye on one screen.
[0028]
DETAILED DESCRIPTION OF THE INVENTION
FIG. 1 is a block diagram showing a configuration of a signal processing system of a digital still camera which is an embodiment of an image processing apparatus according to the present invention. In the figure, reference numeral 1 denotes an objective lens. The light beam incident from the objective lens 1 forms an image on the light receiving surface of the CCD 2 that is an image sensor, and is converted into an analog electric signal corresponding to the brightness. The analog electric signal obtained by the CCD 2 is periodically scanned at an appropriate timing, whereby a signal for one frame is serially transferred to the analog / digital (A / D) converter 3. The A / D converter 3 converts the analog signal inputted serially into a digital signal and sends it to the signal processing circuit 4. The signal processing circuit 4 performs appropriate processing on the serially input digital signal to generate bitmap format image data and sends it to the selector 5.
[0029]
In addition to the signal processing circuit 4 described above, a VRAM 6 and a CPU 7 are connected to the selector 5, and a signal supply source and a supply destination are physically selected and connected in accordance with a selection instruction signal SS given from a system controller (not shown). To do. Specifically, when the display setting for displaying the camera image on the monitor is made, the selector 5 transfers the image data sent from the signal processing circuit 4 to the VRAM 6. If the display setting for displaying an image input from an external device on the monitor is made, the selector 5 transfers the image data sent from the CPU 7 to the VRAM 6. Furthermore, when the shutter is pressed, the selector 5 transfers the image data sent from the signal processing circuit 4 to the CPU 7.
[0030]
By such image data transfer operation by the selector 5, the image data sent to and stored in the VRAM 6 is displayed on the liquid crystal monitor 8 as a finder. The bitmap image data stored in the VRAM 6 is converted into an analog NTSC signal by a digital / analog (D / A) converter 9 and output from a television (TV) terminal 10. Therefore, when the TV monitor 11 is connected to the TV terminal 10, the same image as that displayed on the liquid crystal monitor 8 is also displayed on the TV monitor 11.
[0031]
The CPU 7 compresses the image data in the bitmap format for one frame sent from the selector 5 when the shutter is pressed to the compressed image data for one frame by compressing the data by an appropriate compression method, for example, the JPEG method. To do. The converted data is stored in the image area 12a of the flash memory 12. The data stored in the image area 12a is read by the CPU 7, decompressed to the original image, sent from the selector 5 to the VRAM 6, stored therein, and displayed on the liquid crystal monitor 8 or the TV monitor 11. The flash memory 12 is provided with a program area 12b for storing a program to be described later, and the program is read and executed by the CPU 7.
[0032]
On the other hand, data stored in the image area 12a of the flash memory 12 can be transferred from the PC terminal 13 compliant with the RS-232C interface standard, which is a general personal computer serial terminal, to the personal computer (PC) 14, for example. It is possible. On the other hand, the data compressed by the compression method (JPEG method) can be sent from the PC 14 via the PC terminal 13 and written into the image area 12a of the flash memory 12.
[0033]
FIG. 2 is a flowchart showing a processing procedure relating to inclination correction of the image processing apparatus according to the present invention.
The input of two systems of images is individually accepted (S1). Each image is subjected to DFT to obtain a respective power spectrum, and a power spectrum image representing these in a polar coordinate system is obtained (S2). The luminance level is scanned in a predetermined direction of each power spectrum image, specifically, the vertical direction and the horizontal direction, the peak position is detected, and each line segment connecting the peak positions is set (S3). An angle formed by both line segments is obtained (S4). One image is corrected so that the obtained angle becomes small (S5), and the process is terminated.
[0034]
FIG. 3 is a flowchart showing another processing procedure related to the correction of the inclination of the image processing apparatus according to the present invention.
The input of two systems of images is individually accepted (S11). Each image is subjected to DFT to obtain a respective power spectrum, and a power spectrum image representing these in a polar coordinate system is obtained (S12). The luminance level is scanned in a predetermined direction of each power spectrum image, specifically, the vertical direction and the horizontal direction, the peak position is detected, and each line segment connecting the peak positions is set (S13). An angle formed by both line segments is obtained (S14). One power spectrum image is corrected so that the obtained angle becomes small (S15). By applying IDFT to the corrected power spectrum image, an image with corrected deviation in inclination is obtained (S16), and the process is terminated.
[0035]
FIG. 4 is a flowchart showing a processing procedure related to position correction of the image processing apparatus according to the present invention.
The two images and the number of repetitions are individually received (S21). A rectangular area is cut out near the center of one image (S22). A high-pass filter is applied to the rectangular region cut out of the image by a processing procedure described later to obtain a first inspection image (S23). A cutout position is set in the other image, and the rectangular area is cut out at the cutout position (S24). The high-pass filter is applied to the rectangular area cut out of the image to obtain a second inspection image (S25).
[0036]
The sum of luminance level differences between pixels corresponding to the positions of the first inspection image and the second inspection image is calculated (S26), and the number of iterations is decremented (S27). It is determined whether or not the number of iterations has become 0. If not, the process proceeds to S24 and the subsequent procedure is repeated. In S28, when it is determined that the number of iterations is 0, the pair having the smallest sum is specified (S29). Then, one of the images is corrected so that the distance between the cut-out regions related to the specified set is reduced (S30), and the process is terminated.
[0037]
FIG. 5 is a flowchart showing an example of a processing procedure related to the high-pass filter.
A power spectrum is obtained by applying DFT to the cut out rectangular area (S31). The low frequency component of the power spectrum is removed and IDFT is performed to obtain an inspection image (S32), and the process returns to the caller.
[0038]
The high-pass filter is not limited to the method using the DFT as described above, and an inspection image is obtained using various edge detection operators, for example, a Laplacian operator, which are frequently used in the field of image processing. It may be configured.
[0039]
【The invention's effect】
In the image processing apparatuses according to the third invention as claimed in claim 1 and the fourth invention as claimed in claim 2 as described above, the deviation of the inclination and position of the imaging range between systems is corrected by image processing. It has an excellent effect in simplifying the handling of the camera.
[0040]
Furthermore, since the images or moving images of each system received individually are processed, there is no need to provide a mechanism for simultaneously forming images associated with the right eye and the left eye on one screen. This has an excellent effect on downsizing the apparatus.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of a signal processing system of an image processing apparatus according to the present invention.
FIG. 2 is a flowchart showing a processing procedure relating to inclination correction of the image processing apparatus according to the present invention;
FIG. 3 is a flowchart showing another processing procedure related to inclination correction of the image processing apparatus according to the present invention;
FIG. 4 is a flowchart showing a processing procedure related to position correction of the image processing apparatus according to the present invention.
FIG. 5 is a flowchart illustrating an example of a processing procedure related to a high-pass filter.
FIG. 6 is an explanatory diagram for explaining the concept of the present invention.
FIG. 7 is an explanatory diagram for explaining another concept of the present invention.
FIG. 8 is an explanatory diagram for explaining a shift between imaging ranges of two imaging systems.
[Explanation of symbols]
7 CPU
12 Flash memory
12a Image area
12b Program area

Claims (2)

An image processing apparatus that performs image processing for matching the positions of both subjects with respect to two images obtained by capturing the same subject in order to generate a stereoscopic image,
Means for cutting out a region of a predetermined size at a predetermined position of one image;
Means for applying a high-pass filter to the cut-out region of the image to obtain a first inspection image;
At the predetermined position of the other image and a plurality of positions different from the predetermined position , a region having the same size as the clipped region is cut out at a predetermined number of positions inputted in advance. Means for applying a pass filter to obtain respective second inspection images;
Means for calculating a sum of luminance level differences between pixels corresponding to positions for each set of the obtained first inspection image and each second inspection image;
A means for comparing each calculated sum to identify a pair having a small size;
An image processing apparatus comprising: means for correcting the position of one image so that the difference is reduced based on the difference in the cutout position of each region in the specified group. The high-pass filter is
Means for performing orthogonal transformation on an image cut-out region to obtain a transformation coefficient;
Means for removing low frequency components of the obtained conversion coefficient;
2. An image processing apparatus according to claim 1, further comprising means for performing orthogonal inverse transform on the transform coefficient from which the low frequency component has been removed.

Images