JP4536231B2 - Imaging device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an imaging apparatus that handles stereoscopic images, and more particularly to an imaging apparatus for multi-view stereo images.
[0002]
[Prior art]
There are various proposals for a method of photographing and recording an image such as a subject including three-dimensional information and reproducing and observing the image. Among them, the so-called binocular stereo system that records two images with parallax corresponding to the viewpoints of the left and right eyes and presents them to the left and right eyes is the simplest and cheapest configuration. Has been used since ancient times to today.
[0003]
In this binocular stereo method, there are various display methods. For example, when performing simultaneous observation of a large number of people on a large screen, there are a polarization projection method using polarized glasses and a time-sharing presentation method using shutter glasses. Although these are used, all of them are large and require an expensive system, so they are rarely used except for special business use. Therefore, for so-called personal use, there is a restriction that only one person can observe at the same time, but the method using stereo pair images, which is the most basic and classic method, is very inexpensive and produces clear images. It is still widely used today as an observable method.
[0004]
To describe this stereo pair image in detail, an L image that is an image corresponding to the left eye viewpoint and an R image that is an image corresponding to the right eye viewpoint are usually arranged in parallel with a small gap between two images. It is composed. The LR image is actually one standard horizontal position image (horizontal 3: vertical) due to restrictions on the system of 35 mm single-lens reflex camera + stereo adapter, which is popular as the most convenient image capturing device of this kind. 2) is vertically divided into two, and therefore, each image, that is, the observed stereoscopic image is generally in a horizontal position (aspect ratio of about 3: 4).
[0005]
In the present specification, an image in which LR2 images are spatially arranged on the image plane to form one image is referred to as a stereo pair image. Note that the specific configuration (numerical values) described above is merely an example, but for the sake of simplicity, the description will be made on the assumption that the specific example is taken as an example unless otherwise specified.
[0006]
This stereo pair image
(1) No special system is required for recording, printing, transmission, printing, etc.
(2) Direct observation is possible if appropriate conditions are met. That is, the left and right images can be fused without using any device.
It has an extremely excellent feature.
[0007]
In particular, with respect to (2), a state in which the LR image is correctly captured by the left and right eyes and recognized as one stereoscopic image instead of two different images is referred to as fusion of the left and right images. In the case of “parallel arrangement” (specifically, the width is slightly smaller than twice the eye width = 10 to 13 cm) (L is placed on the left and R is placed on the right), the line of sight is also observed during the observation. Fusion is possible by using the so-called “parallel method” (which requires some practice depending on the person). In addition, the “intersection arrangement” in which the left and right images are interchanged is also used, and there is no restriction on the print size, and it is possible to observe directly by the “intersection method” that intersects the lines of sight. Since the unnaturalness (a miniature garden phenomenon) at the time of stereoscopic observation is somewhat large, the parallel arrangement is more popular.
[0008]
In any case, stereo pair images with the two major features of (1) regardless of system (2) direct observation are possible in this way, especially if so-called media mixing is promoted, such as the spread of the Internet and digital cameras. It is expected that the value will be reviewed and continue to be used as it progresses.
[0009]
Therefore, the applicant of the present invention has previously observed an observation method and an observation that give different shifts to the LR image in order to easily perform high-quality observation with high presence using SPM (Stereo Pear in Multimedia) which is an electronic stereo pair image. A technology relating to a device (viewer) has been proposed (Japanese Patent Application No. 2000-115357). In this technique, the same image is usually displayed on the two display devices (monocular mode). However, when the SPM is read from the memory card and displayed, the left eye image side and the right eye image side are displayed. Different shift processes are performed by the stereo pair image shift processing unit, and the image signals after the shift process are displayed as left eye images and right eye images from the display device (stereo mode). By this shift processing, the L image of the SPM on the left eye image side and the R image of the SPM on the right eye image side are merged as one stereoscopic image, and a high-quality stereoscopic image with a high sense of reality can be easily obtained. It becomes possible to observe.
[0010]
On the other hand, as a digital still camera (electronic camera) for capturing a stereo image, for example, there is one described in Japanese Patent Laid-Open No. 5-30538 invented by the present inventor. This obtains an LR image using two image pickup systems (lens + image pickup device), and adds stereo pair code data (data indicating a stereo image, identification information of L and R, etc.) to each image individually. It is excellent because it can be easily recorded in stereo.
[0011]
[Problems to be solved by the invention]
However, the electronic camera disclosed in the above publication has the following inadequate points.
(a) Since two imaging systems are required, an increase in size and price is inevitable.
(b) Since the obtained stereo image is different from the SPM (stereo pair arranged in parallel), a separate process is required to obtain the SPM.
[0012]
Here, if a conventional (monocular) digital camera is used in combination with a conventional stereo adapter, it seems that both (a) and (b) above are solved.
(c) No stereo data is included, including that the image is a stereo image. Therefore, the management becomes complicated in a mixed situation with a normal flat (non-stereo) image, and an application for automating image shift at the time of reproduction in the viewer as in the prior application becomes impossible.
[0013]
In particular, for this problem, as described in the above publication, it can be considered that a stereo pair code is added to each image file in which L and R images are recorded. In the case where a stereo adapter is combined with a camera, each of the L and R images cannot be handled as an individual image.
[0014]
In addition, (d) when an external stereo adapter is added to the image pickup system of a normal monocular camera, there is a problem that troubles such as overlap and vignetting peculiar to the adapter method cannot be avoided. FIG. 5A shows a case where the F value is small (on the open side), and LR images overlap. FIG. 5B shows a case where the F value is large (on the narrowing side), and vignetting occurs between the LR images.
[0015]
The present invention has been made in consideration of the above circumstances, and the object of the present invention is to easily capture a multi-view stereo image to which high-quality stereo data is added and to connect a stereo adapter. An object of the present invention is to provide an imaging apparatus capable of avoiding the effects of various malfunction phenomena when used.
[0016]
[Means for Solving the Problems]
(Constitution)
In order to solve the above problems, the present invention adopts the following configuration.
[0017]
That is, the present invention provides an imaging device for capturing a multi-view stereo image, an image sensor that photoelectrically converts a subject image formed by an imaging optical system, and a front stage of the imaging optical system, and according to parallax. A mirror-type stereo adapter that receives light from the subject at different positions and guides the light to different regions of the imaging device, an imaging unit that obtains a subject image signal based on the output of the imaging device, and the subject image signal A photographic image frame that sets a plurality of photographic image frames corresponding to a plurality of monocular images that are constituent elements of one multi-view stereo image in the imaging region of the imaging device by performing predetermined trimming on the image sensor. and setting means, a plurality of monocular images obtained in correspondence with the plurality of photographing frame, by arranging the different positions regions on one image, the parallel arrangement type is one planar image Image synthesizing means for synthesizing a teleo image; and recording means for recording the parallel arrangement type stereo image. The image synthesizing means is configured to combine the parallel arrangement type stereo image when synthesizing the parallel arrangement type stereo image. An image frame image representing a frame line is superimposed on a boundary region of each monocular image on the image or around the parallel stereo image .
[0018]
Here, preferred embodiments of the present invention include the following.
[0020]
( 1 ) A plurality of monocular images are L and R images which are components of a binocular stereo image, and a parallel arrangement type stereo image synthesized by the image synthesis means is a stereo pair in which L and R images are arranged on the left and right. It must be an image.
[0022]
(Function)
According to the present invention, since a plurality of monocular image frames are set in the imaging region of one image sensor, and a stereo image is generated from the plurality of monocular images obtained from the frame, a multi-view stereo image is captured. Therefore, a plurality of image sensors are not required, and a stereo adapter and one image sensor are sufficient. That is, the device configuration can be reduced in size and price. In addition, by appropriately setting the monocular image frame, it is possible to avoid problems such as overlap and vignetting peculiar to the adapter method.
[0023]
DETAILED DESCRIPTION OF THE INVENTION
The details of the present invention will be described below with reference to the illustrated embodiments.
[0024]
FIG. 1 is a block diagram showing a circuit configuration of an electronic camera according to an embodiment of the present invention.
[0025]
In the figure, 101 is a lens system composed of various lenses, 102 is a lens driving mechanism for driving the lens system 101, 103 is an exposure control mechanism including an aperture and a shutter of the lens system 101, and 104 is a filter system such as an optical low-pass filter. 105 is a CCD color image sensor, 106 is a CCD driver for driving the image sensor 105, 107 is a preprocess circuit including an A / D converter, 108 is a color signal generation process, matrix conversion process, and other various digital A digital process circuit for performing processing, 109 is a card interface, 110 is a memory card such as a CF, and 111 is an LCD image display system.
[0026]
Also, 112 in the figure is a system controller (CPU) 112 for overall control of each part, 113 is an operation switch system composed of various SWs, 114 is an operation display system for displaying operation states and mode states, 115 Is a lens driver for controlling the lens driving mechanism 102, 116 is a strobe as a light emitting means, 117 is an exposure control driver for controlling the exposure control mechanism 103 and the strobe 116, and 118 is for storing various setting information and the like. A non-volatile memory (EEPROM) is shown.
[0027]
In the electronic camera according to the present embodiment, the system controller 112 performs overall control, and controls exposure (charge accumulation) and signals by controlling the driving of the CCD image sensor 105 by the exposure control mechanism 103 and the CCD driver 106. Is read into the digital process circuit 108 via the pre-process circuit 107 and subjected to various signal processing, and then recorded on the memory card 110 via the card interface 109. The CCD image sensor 105 is, for example, an interline type having a vertical overflow drain structure.
[0028]
The basic configuration so far is the same as that of a conventional general electronic camera, but in this embodiment, in addition to this, the operation switch system 113 is provided with a mode switch for switching between the normal mode and the stereo mode. It has been. Further, the system controller 112 includes an image frame setting unit for setting a trimming area of an imaging area, an SPM synthesis unit for synthesizing an SPM image from a monocular image, and an SGM generation for generating JPEG image data from the SPM image. Is provided.
[0029]
In the electronic camera of this embodiment, as shown in FIG. 2, a mirror stereo adapter 200 can be attached to and detached from the camera body 100. The adapter 200 is configured by disposing mirrors 201 and 202 at positions separated by a parallax, and further disposing mirrors 203 and 204 for guiding light reflected by the mirrors 201 and 202 to the camera side. Light incident on the right mirror 201 of the adapter 200 forms an image on the left region L of the image sensor 105 via the mirror 203 and the photographing lens 101, and light incident on the left mirror 202 passes through the mirror 204 and the photographing lens 101. Thus, an image is formed on the right region R of the image sensor 105.
[0030]
In the normal mode, an image is taken in the same manner as a normal (monocular) camera without attaching the adapter 200. Note that switching between the normal mode and the stereo mode is performed by the mode switch described above, but it may be automatically switched according to whether or not the mirror type stereo adapter 200 is attached.
[0031]
In the stereo mode, the same imaging as that of a normal camera is performed by a trigger operation with the stereo adapter 200 attached. Then, the following processing is performed on the image signals read from the entire imaging area.
[0032]
First, trimming is performed in the digital process circuit 108 by an image frame setting unit included in the system controller 112. That is, as shown in FIG. 3, the screen is vertically divided into two, and the left half is assigned as an L image and the right half is assigned as an R image (100% trimming). Although it is possible to use 100% trimming, in this embodiment, the horizontal width is limited to 90% in order to prevent overlap and vignetting from appearing in the image, and further (although it is not essential) the vertical and horizontal balance is adjusted. For this purpose, the trimmed images with the same vertical width are assigned to the L and R images, respectively.
[0033]
Next, an SPM image is generated in the digital process circuit 108 by an SPM synthesis unit included in the system controller 112. That is, the L and R images obtained by the above trimming are combined as shown in FIG. 4 and become one image (SPM image) in which two images are arranged in parallel on the left and right without any gap. At this time, a border line having a width of 1 to several pixels (for example, black of R = G = B = 0) is arranged in the boundary region or around the SPM image, so that it is visually clear that the image is an SPM image. It is also a preferable modification example.
[0034]
Then, the SPM image is generated by the SGM (Stereo Gram in Multimedia) generation unit included in the system controller 112 into JPEG image data added with stereo data as a header portion. That is, it is preferable to compress the image information during recording and transmission, and any method can be used at that time, but the most standard known JPEG compression is used. At that time, for example, stereo information is assigned to a predetermined tag in the user information area of the header portion. The information to record is
a: Stereo or not (default: Y)
b: Number of images in stereo (default: 2)
c: Arrangement of each monocular image (existing area including the number of vertical and horizontal pixels)
Is basic information.
[0035]
If these pieces of information are present, the device itself or another device can separate and reproduce each original monocular image by reading this information (the image may be cut out according to the number and arrangement information after the image is expanded). ). Such an SGM is an example of a (stereoscopic) structured stereo image because all the image data of one stereo image and necessary stereo data other than the image data are used as one file, that is, a handling unit. is there. Since it is an inseparable handling unit for general use, for example, on a general-purpose PC or for transmission on the Internet, there is no problem that only some of them are erroneously recorded, transmitted, or deleted.
[0036]
However, in the present invention, the structured stereo image is a broad definition. Therefore, in addition to the above example, the SGM may be configured such that the image data as shown in the conventional example is divided into a plurality of files. That is, it is self-evident that effects such as reduction in size and weight and avoidance of problems when using an adapter can be obtained in the same manner regardless of the SGM format. Note that the above-described trimming (cutting out L and R images from one captured image frame) is indispensable in order to configure such an SGM having a plurality of file formats.
[0037]
The generated SGM is recorded on the memory card 110 via the card interface 109 by a recording means in the digital process circuit 108 that operates according to an instruction from the system controller 112.
[0038]
Note that the card on which the SGM is recorded is used by being replaced with a slot of a general-purpose PC, for example. The camera body also has an input / output port, and the generated SGM can be input / output by wired or wireless connection. It is also possible to input / output SGM via a card (card interface).
[0039]
As described above, according to the present embodiment, the mirror type stereo adapter 200 is attached to the electronic camera 100 having the same configuration as that of a normal monocular electronic camera, and a plurality of monocular image frames are formed on the imaging area of one imaging element 105. Since a stereo image is obtained from a plurality of monocular images obtained from this, a plurality of image sensors are not required. For this reason, size reduction and price reduction are possible. Further, by appropriately setting the image frame, it is possible to avoid problems such as overlap and vignetting peculiar to the adapter method.
[0040]
In addition, this invention is not limited to embodiment mentioned above. In the embodiment, SPM, SGM, and the like have been exemplified for a still image, but the present invention is not limited to this, and can be applied in the same manner even for a moving image. That is, since the SPM composition method and the stereo data added at the time of SGM generation include only elements that are not different from still image moving images, the conventional moving image imaging technology is used as it is, for example, a typical moving image compression format. By using MPEG, which is one of the above, instead of the above-mentioned JPEG, the same effect can be obtained by carrying out in exactly the same way.
[0041]
In the embodiment, two image frames are set in the imaging area. For example, a general multi-view stereoscopic image used in a known lenticular method (such as a multi-projection method using a lenticular sheet print or a lenticular screen) is obtained. Therefore, three or more mirror adapters and three or more image frame settings corresponding thereto may be used. In addition, various modifications can be made without departing from the scope of the present invention.
[0042]
【The invention's effect】
As described above in detail, according to the present invention, a stereo image is picked up by a single image pickup device using a mirror type stereo adapter, and a plurality of shooting image frames corresponding to a monocular image are provided in the image pickup region of the image pickup device. Since a stereo image is generated from a plurality of monocular images set and obtained from this, it is possible to reduce the size and cost of the apparatus without requiring a plurality of imaging elements for stereo imaging. In addition, by appropriately setting the image frame, it is possible to avoid problems such as overlap and vignetting peculiar to the adapter method.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a circuit configuration of an electronic camera according to an embodiment of the present invention.
FIG. 2 is a diagram showing a configuration of a mirror type stereo adapter attached to the electronic camera of the embodiment.
FIG. 3 is a diagram showing an image frame set in an imaging area.
FIG. 4 is a diagram showing a state in which an SPM image is generated by combining L and R images, and further an SGM is generated.
FIG. 5 is a diagram showing overlap and vignetting peculiar to an adapter method, which is a conventional problem.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 100 ... Camera body 101 ... Lens system 102 ... Lens drive mechanism 103 ... Exposure control mechanism 104 ... Filter system 105 ... CCD color image sensor 106 ... CCD driver 107 ... Pre-process part 108 ... Digital process part 109 ... Card interface 110 ... Memory card 111 ... LCD image display system 112 ... System controller (CPU)
113 ... Operation switch system 114 ... Operation display system 115 ... Lens driver 116 ... Strobe 117 ... Exposure control driver 118 ... Non-volatile memory (EEPROM)
200 ... Mirror type stereo adapter 201-204 ... Mirror

Claims (2)

An image sensor that photoelectrically converts a subject image formed by the imaging optical system;
A mirror-type stereo adapter that is arranged in a front stage of the imaging optical system, receives light from the subject at different positions according to parallax, and guides the light to different areas of the imaging device;
An imaging unit that obtains a subject image signal based on the output of the imaging device, and performing predetermined trimming on the subject image signal, thereby allowing one multi-view stereo image to be captured in the imaging region of the imaging device. Photographing image frame setting means for setting a plurality of photographing image frames corresponding to a plurality of monocular images as constituent elements;
Image synthesizing means for synthesizing a parallel arrangement type stereo image which is one planar image by arranging a plurality of monocular images obtained corresponding to the plurality of photographed image frames in different position areas on one image. And
Recording means for recording the parallel arrangement type stereo image;
Comprising
The image synthesizing means, when synthesizing the parallel arrangement type stereo image, an image frame image representing a frame line in a boundary region of each monocular image on the parallel arrangement type stereo image or further around the parallel arrangement type stereo image An imaging apparatus characterized by being configured to superimpose . The plurality of monocular images are L and R images which are components of a binocular stereo image, and the parallel arrangement type stereo image synthesized by the image synthesizing unit is a stereo pair in which the L and R images are arranged on the left and right. The imaging apparatus according to claim 1 , wherein the imaging apparatus is an image.

Images