JP5725758B2 - Imaging device - Google Patents

Description

  The present invention relates to an imaging apparatus that performs moving image shooting such as a video camera or a digital camera.

  In recent photographing apparatuses (cameras), research for enhancing a main subject by blurring a background portion of an image at the time of a still image is underway. For example, in Patent Document 1, a background region is extracted by comparing two images with different photographing apertures, and a blurring effect is obtained by image processing for the region.

JP 2001-157107 A

  The technique disclosed in Patent Document 1 described above is applied to conditions that require only one image, such as still image shooting. However, in moving images, there are problems of quality and reliability due to the necessity to frequently open and close the aperture, and an increase in image processing load.

(Object of invention)
An object of the present invention is to provide an imaging apparatus capable of obtaining a background blur effect without reducing reliability and increasing image load even in moving image shooting.

In order to achieve the above object, an imaging apparatus according to the present invention includes an imaging unit that captures a subject, a determination unit that determines a main subject region and a background region in an image captured by the imaging unit, and the determination unit. A background image processing unit that performs a blurring process on the determined background region, a composition change detection unit that detects a composition change of the photographing unit, and the background unit is set by operating the determination unit at the start of moving image shooting. After that, only when the composition change is detected by the composition change detection means, the determination means is operated to reset the background area, and a moving image is obtained from a plurality of images blurred by the background image processing means. And a moving image creating means for creating the image.

  According to the present invention, it is possible to obtain a background blur effect without reducing reliability or increasing image load even in moving image shooting.

1 is a block diagram illustrating a photographing apparatus that is Embodiment 1 of the present invention. FIG. FIG. 2 is a block diagram illustrating an imaging unit and a discrimination control unit in Embodiment 1. It is a block diagram which shows the discrimination means in Example 1. FIG. It is a figure explaining the background blurring process of Example 1. FIG. It is a figure explaining the moving image creation of Example 1. FIG. It is a figure explaining another system of the background blurring process of Example 1. FIG. 6 is a block diagram illustrating an imaging apparatus that is Embodiment 2. FIG. It is a figure explaining the background blurring process of Example 2. FIG. It is a figure explaining the background blurring process of Example 3. FIG. It is a figure explaining the background blurring process of Example 4. FIG. FIG. 10 is a diagram illustrating a configuration of an image sensor in Example 5.

  The mode for carrying out the present invention is as shown in Examples 1 to 5 below.

  FIG. 1 is a block diagram of an image pickup apparatus in Embodiment 1 of the present invention. In FIG. 1, reference numeral 101 denotes an imaging unit including an imaging optical system, an image sensor, elements such as an imaging aperture, a shutter, and a strobe and a control unit that controls them. The image photographed by the photographing means 101 is input to the discrimination means 103, and the main subject area and the background area in the obtained image are separated.

  Here, there are various methods for separating the main subject and the background, but in the first embodiment, this is performed based on the difference between two images obtained by changing the photographing aperture. Although the details of the operation will be described later, the point of the present invention is that the discrimination means 103 is controlled by the discrimination control means 102 so as to operate only at the start of photographing and at the time of composition change.

  The discrimination control unit 102 controls the operation of the discrimination unit 103 in accordance with the shooting state of the shooting unit 101. Details regarding this control will be described later.

  When the determination unit 103 separates the main subject region and the background region in the image, the background image processing unit 104 performs a blurring process on the background region of the image obtained by the photographing unit 101 based on the information. There are various methods for the blurring process. For example, there is a method in which a new pixel obtained by weighting and adding the luminance of the target background pixel and the luminance of the pixel in the vicinity of the target pixel is used as the luminance of the target pixel. is there.

  The moving image creating means 105 joins a plurality of background-blurred images arranged as described above, which are obtained as described above, to form a moving image. With the above configuration, a moving image with a blurred background can be created.

  Here, the details of the operation of the discrimination control means 102 will be described with reference to FIG. In FIG. 2, the discrimination control means 102 is provided with a composition change detection unit 102a and a discrimination operation control unit 102b. The composition change detection unit 102a is a motion vector of an image obtained from imaging information of the image sensor 101a provided in the imaging unit 101, and a vibration that is an output of a vibration detection unit 101b such as an angular velocity meter provided in the imaging unit 101. A change in the shooting composition is detected based on the information. For example, when the motion vector is detected but there is no output from the vibration detection unit 101b, it is detected that the subject is moving and there is no change in the shooting composition, and in the opposite case, the tracking shooting is determined. When both outputs occur at the same timing, it is caught and detected as a composition change. Of course, from the viewpoint of simplification of the system, either the motion vector or the output of the vibration detection unit 101b may be captured and detected as a composition change. When the composition change detection unit 102a detects a composition change, the determination operation control unit 102b causes the determination unit 103 to perform a determination operation.

  Further, when the photographing start operation unit 101 c provided in the photographing unit 101 is operated and photographing is started, the signal is output to the photographing start detection unit 102 c of the discrimination control unit 102. When the imaging start detection unit 102c detects the start of imaging, the determination operation control unit 102b causes the determination unit 103 to perform a determination operation.

  In this way, since the determination unit 103 is operated only at the beginning of photographing and when the composition is changed, the load for separating the main subject from the background can be greatly reduced.

  FIG. 3 is a diagram for explaining the details of the discrimination means 103. The determination unit 103 includes a tracking unit 103a, a background separation unit 103b, and an image quality changing unit 103c. The background separation unit 103b separates the main subject and the background based on the tracking information of the tracking unit 103a and the image quality change timing from the image quality changing unit 103c.

  The background separation method in Embodiment 1 will be described below. The image quality changing unit 103c changes the photographing aperture included in the photographing optical system of the photographing unit 101 to capture a plurality of images, for example, two images. For example, the image of the shooting aperture F4.0 and the image of the shooting aperture F13 are captured without a time interval, and the images are captured. In FIG. 4, 401 is an image taken at F4.0, 402 is an image for background recognition taken at F13, and the background blur is different.

  Note that these images 401 and 402 are taken with the same exposure time in order to make the influence of the subject shake the same. For this reason, the image 402 of F13 is underexposed compared to the image 401 photographed at F4.0. In order to correct this, the gain of the image is increased so that the image 402 of F13 has the same brightness as the image 401 of F4.0.

  An image 403 is obtained by aligning the images thus obtained based on the difference between the motion vectors. Here, the alignment and composition are performed in order to eliminate the influence of camera shake during the photographing of two images.

  The background separation unit 103b recognizes a region (blur changing unit) where there is a difference in blur in the image 403 by comparing the image quality of the images 401 and 402 as a background, and performs contour extraction with the main subject. Then, the background-extracted information is output to the background image processing means 104. The background extraction information of the background separation unit 103b is also input to the tracking unit 103a. The tracking unit 103a updates the movement information of the main subject from the contour information obtained by the background separation unit 103b and the motion vector of the photographing information of the photographing unit 101, and outputs the information to the background separation unit 103b. Specifically, the motion vector in the contour region is output. The background separation unit 103 b updates the contour information based on the motion vector of the tracking unit 103 a and outputs the updated outline information to the background image processing unit 104. As described above, the background image processing unit 104 performs a blurring process on the background area. Thereby, an image 404 is obtained. Then, the moving image creating means 105 joins a plurality of background-blurred images, which are obtained as described above, arranged side by side in time to form a moving image.

  FIG. 5 illustrates the image situation after the background image processing, and when the first image at the start of photographing or the composition change or panning of the camera is performed, an image 501s obtained by reducing the aperture and increasing the gain as a sub image is acquired. To do. The image 501s is compared with the normal aperture (F4.0) image 501 to obtain an image 501a with a blurred background by the method described above. After that, the background of the images 502, 503,... Is blurred while tracking the main subject (contour) to create the images 502a, 503a,. Then, the created images are continued to obtain a moving image.

  As described above, since the background area is extracted only at the start of photographing and when the composition is changed, it is possible to greatly reduce the calculation load and the mechanical load for driving the aperture. In FIG. 4, contour extraction is performed by comparing the image 401 and the image 402, and background blurring processing is performed on the image 401 based on the result. However, in addition to such a method, two images having a size smaller than the actually recorded image are prepared by changing the aperture, and the background blur processing is performed on the recorded image based on the contour information obtained from the comparison. May be performed.

  FIG. 6 is a diagram for explaining the blurring process described above. Two images 602 and 603 with different apertures are taken apart from an image 601 used for moving image shooting (exposure adjustment of each other image). Do). Then, the background portions are extracted as shown in the image 604 by comparing the images 602 and 603 after alignment. Based on the contour information obtained as a result, the background of the image 601 is blurred by the background image processing means 104 to obtain an image 605. Then, a moving image is obtained continuously with an image acquired thereafter.

  In the first embodiment, the image quality changing unit is a diaphragm control unit that changes the size of the photographing diaphragm for each photographed image.

  With the above configuration, it is possible to obtain a moving image with a blurred background with a light load.

  A second embodiment of the present invention will be described with reference to FIGS. The difference between the second embodiment and the first embodiment is the difference in the image quality changing unit 103c in the determination unit 103, and the difference will be described below.

  In the first embodiment, the image quality is changed by changing the aperture. This is because the background blur is changed by changing the aperture, and the background portion is extracted by comparing two images whose background blur is changed. In the second embodiment, the aperture is not changed, but the brightness of the main subject of the image that illuminates the subject and the image that is not illuminated is compared to separate the background.

  FIG. 7 is a diagram for explaining the details of the photographing unit 101 and the image quality changing unit 103c. Reference numeral 201 denotes a photographing optical system, which forms a subject image on the image sensor 201f by a known bending optical system 201a. Reference numeral 201b denotes an aperture controlled by the aperture control unit 201h. Reference numeral 201c denotes an illumination light source which is an illumination unit such as a strobe or an LED which is controlled by the illumination control unit 201g to project illumination on the subject. Reference numeral 201d denotes a pattern film that changes the illumination light of the illumination light source 201c into a predetermined pattern. The pattern film 201d is inserted during creation of a recorded image so that the projection of the illumination light source 201c has no pattern. Have been).

  Reference numeral 201e denotes a shake correction optical system that is controlled by the image stabilization control unit 201i to reduce shake applied to the photographing unit 101 by moving in the direction of the arrow.

  FIG. 8 is a diagram for explaining image quality change using the illumination light source 201c and the pattern film 201d. Reference numeral 801 denotes a recorded image. After the recorded image is acquired, two smaller-sized images 802 and 803 are acquired for main subject recognition. The image 802 is similar to the image 801 resized, and the image 801 may be resized and used. Reference numeral 803 denotes an image using the illumination light source 201c and the pattern film 201d. The main subject close to the imaging apparatus is illuminated with pattern projection, but the pattern projection does not reach the background. The intensity of the illumination light source 201c is dimmed by making it weaker as the distance to the main subject is shorter. However, when the determination unit 103 is operated, the background is not illuminated by using weaker illumination. .

  When the images 802 and 803 are aligned and compared, the brightness changes only in the main subject area (because it is illuminated), so the main subject and the background can be separated based on the information. Although the illumination light is weakened, it is weak against disturbance noise. However, because of the pattern projection, the detection accuracy of the illumination area is improved by extracting the frequency component.

  The background image processing unit 104 is operated based on the main subject and background contour information obtained from the image 804 to blur the background as in the image 805, and the subsequent image is used as the contour information of the image 804 and the tracking information of the tracking unit 103a. Based on this, the background is processed.

  The operation of the discriminating means 103 is the same as in the first embodiment at the start of photographing and at the time of changing the composition, and since it is not operated otherwise, the illumination projection does not continue, so that not only power saving but also photographing is performed. You can reduce the sense of discomfort on the side.

  As described above, the image quality changing unit 103c includes the illumination unit 201c that irradiates the subject with auxiliary light for each captured image and the illumination control unit 201g that changes the illumination light amount of the illumination unit 201c for each captured image. And the background and the main subject are separated by changing the image quality of the two images by projecting a predetermined pattern on the photographic subject.

  FIG. 9 is a diagram for explaining image quality change in the third embodiment of the present invention. The difference between the third embodiment and the first embodiment is a difference in the image quality changing unit 103c in the determination unit 103, and the difference will be described below. In the first embodiment, the image quality is changed by changing the aperture. This is because the background blur is changed by changing the aperture, and the background portion is extracted by comparing two images whose background blur is changed. In the third embodiment, the main subject and the background are separated using the shift of the photographing pupil position instead of changing the aperture.

  In FIG. 9, reference numeral 901 denotes a recorded image, and two images 902 and 903 of smaller sizes are acquired for main subject recognition after the recorded image is acquired. The image 902 is similar to the image 901 resized, and the image 901 may be resized and used.

  Reference numeral 902 denotes an image taken by controlling the shake correction optical system 201e in FIG. 7 by the image stabilization control unit 201i so as to be positioned at the leftmost end (left pupil). Similarly, reference numeral 903 denotes an image taken by controlling the shake correcting optical system 201e by the image stabilization control unit 201i and being positioned at the right end (right pupil). When comparing the images 902 and 903 with the main subject in focus being aligned, the left and right backgrounds are shifted due to the difference in pupils. Therefore, the main subject and the background can be separated by recognizing the area where the deviation has occurred as the background.

  The background image processing means 104 is operated based on the main subject and background contour information obtained from the image 904 to blur the background as in the image 905, and the subsequent image is used as the contour information of the image 904 and the tracking information of the tracking unit 103a. Based on this, the background is blurred.

  The operation of the discriminating means 103 is at the start of photographing and at the time of changing the composition as in the first embodiment. Since it is not operated otherwise, the shake correction optical system 201e is not continuously moved, so that only power saving is achieved. In addition, it is possible to reduce the sense of incongruity on the shooting side.

  As described above, the image quality changing unit separates the background and the main subject by providing a pupil position changing unit that changes the optical pupil position of the imaging unit for each captured image.

  FIG. 10 is a diagram for explaining background separation in the fourth embodiment of the present invention. Until the third embodiment, the background and the main subject are separated from each other by comparing two images with different image quality. On the other hand, although the accuracy is inferior, there is a method of using the contrast evaluation value of the image as a method of separating the background and the subject more easily. Since the background area in the image is not in focus, the contrast is slightly lower than that of the main subject. Therefore, the determination unit 103 may operate the background image processing unit 104 by recognizing a low contrast area as the background.

  In FIG. 10, reference numeral 1001 denotes a recorded image. In the recorded image 1001, the main subject is in focus (in focus), and the background is out of focus (out of focus). In the image 1002, the discriminating means 103 calculates the contrast evaluation value of this image and extracts a region having a low evaluation value as the background. The background image processing unit 104 is operated based on the main subject and background contour information obtained from the image 1002 to blur the background as in the image 1003, and the subsequent image is used as the contour information of the image 1002 and the tracking information of the tracking unit 103a. Based on this, the background is blurred.

  The operation of the discriminating means 103 is at the start of shooting and at the time of composition change as in the first embodiment. Since the operation is not performed otherwise, the contrast evaluation value calculation does not continue. This can reduce the discomfort on the side.

  As described above, the discriminating unit 103 discriminates the main subject region and the background region in the image based on the contrast evaluation value of the image formed on the image sensor 101a by the photographing unit 101.

  FIG. 11 is a diagram illustrating the configuration of an image sensor that performs background separation in Embodiment 5 of the present invention. In the third embodiment, the main subject and the background are separated using the shift of the photographing pupil position instead of changing the aperture. As a means for that purpose, the shake correction optical system 201e is moved left and right to acquire two images.

  In the fifth embodiment, the photographing pupil is displaced by the structure of the image sensor.

  FIG. 11A illustrates an image sensor 1101. The image sensor 1101 includes a plurality of photoelectric conversion elements. Here, reference numeral 1102a denotes a Bayer unit in which four photoelectric conversion elements are each covered with an RGB color filter to form a so-called Bayer arrangement, and these units are two-dimensionally arranged in order from 1102b.

  FIG. 11B is an enlarged view of the Bayer unit 1102a. 1103a, 1103b, 1103c, and 1103d are photoelectric conversion elements of the respective color filters, and 1104a, 1104b, 1104c, and 1104d are photoelectric conversion elements having a smaller size than the photoelectric conversion elements 1103a to 1103d. 1107a and 1107d are microlenses formed on the G (green) color filter, 1107b is a microlens formed on the R (red) color filter, and 1107c is a microlens formed on the B (blue) color filter. It is a lens. They are arranged so as to straddle the large and small photoelectric conversion elements 1103 and 1104, respectively. Reference numeral 1105 denotes a mask for separating light into each photoelectric conversion element. Holes 1105a, 1106a, 1105b, 1106b, 1105c, 1106c, 1105d, and 1106d are provided on each photoelectric conversion element and enter each other. The structure is such that the subject luminous flux does not interfere. Note that a dedicated mask may be arranged for the mask 1105, but it may be constituted by a layout of wiring to each photoelectric conversion element.

  In the above structure, images are formed using the outputs of the photoelectric conversion elements 1103a, 1103b,. As in the third embodiment, two images for subject extraction are an image A formed by the outputs of the photoelectric conversion elements 1103a, 1103b,... And an image B formed by the photoelectric conversion elements 1104a, 1104b,. Make a comparison.

  Thereafter, the operation is the same as that shown in FIG. 9, and image A becomes images 901 and 902, and image B becomes image 903. However, since the photoelectric conversion elements 1104a, 1104b,... That form the image B are smaller than the photoelectric conversion elements 1103a, 1103b,. Therefore, the gain of the image B is increased, and the brightness is the same as that of the image A.

  When comparing images A and B with the main subject in focus being compared, the left and right backgrounds are displaced due to pupil differences. This is because the photoelectric conversion elements 1103 and 1104 shifted in the left and right directions are provided in one microlens 1107, so that the photographing pupil is equal to the shifted state. Therefore, the main subject and the background can be separated by recognizing the area where the deviation has occurred as the background.

  The background image processing unit 104 is operated based on the main subject and background contour information obtained from the images A and B to blur the background, and the subsequent images are based on the contour information of the images A and B and the tracking information of the tracking unit 103a. To blur the background.

  The operation of the discriminating means 103 is at the start of photographing and at the time of changing the composition as in the first embodiment. Since the operation is not performed otherwise, the comparison between the images A and B is not continuous. , The sense of incongruity on the side of shooting can be reduced.

  As described above, the discrimination means 103 separates the main subject area and the background area in the image based on the imaging signal correlation values of the images formed on the different areas (photoelectric conversion elements 1103 and 1104) of the imaging element 1101. ing.

  Although the present invention has been described with reference to video camera video applications, it can also be widely applied to digital camera video modes, surveillance cameras, web cameras, mobile phones, and the like.

DESCRIPTION OF SYMBOLS 101 Image | photographing means 102 Discrimination control means 103 Discrimination means 104 Background image processing means 105 Moving image creation means

Claims (17)

Photographing means for photographing the subject;
Discriminating means for discriminating a main subject area and a background area in an image photographed by the photographing means;
Background image processing means for performing a blurring process on the background area determined by the determination means;
Composition change detecting means for detecting composition change of the photographing means;
A discriminating control unit for operating the discriminating unit to reset the background region only when the composition change is detected by the composition change detecting unit after the discriminating unit is operated at the start of moving image shooting and the background region is set ; ,
An imaging apparatus comprising: a moving image creating unit that creates a moving image from a plurality of images subjected to the blurring process by the background image processing unit.   The imaging apparatus according to claim 1, wherein the determination control unit operates the determination unit at an initial stage when shooting by the shooting unit is started.   The imaging apparatus according to claim 1, wherein the determination control unit operates the determination unit after the composition by the photographing unit is changed.   The imaging apparatus according to claim 3, wherein the discrimination control unit further includes a composition change detection unit that detects a composition change based on photographing information of the photographing unit.   The imaging apparatus according to claim 3, wherein the determination control unit further includes a composition change detection unit that detects a composition change based on vibration information of a vibration detection unit included in the photographing unit.   The imaging apparatus according to claim 3, wherein the discrimination control unit further includes a composition change detection unit that detects a composition change based on photographing information of the photographing unit and vibration information of the vibration detecting unit. The determining means further comprises tracking means for tracking the main subject area or the background area based on the determined information of the main subject area and the background area,
7. The blurring process according to claim 1, wherein after the main subject area and the background area are discriminated, the background image processing unit performs blurring processing based on an output of the tracking unit. Imaging device.   2. The discriminating unit includes an image quality changing unit that acquires a plurality of images having different image quality, and a background separating unit that separates the background region by comparing the image quality with each other. 8. The imaging device according to any one of items 7 to 7.   The moving image creating unit creates a moving image from an image subjected to blurring processing by the background image processing unit using any one of a plurality of images acquired by the image quality changing unit. 8. The imaging device according to 8.   The moving image creating means creates a moving image from an image subjected to blurring processing by the background image processing means using an image different from the plurality of images acquired by the image quality changing means. Item 9. The imaging device according to Item 8.   The imaging apparatus according to any one of claims 8 to 10, wherein the image quality changing unit includes an aperture control unit that changes a size of an imaging aperture for each captured image.   The imaging apparatus according to claim 8, wherein the image quality changing unit includes a pupil position changing unit that changes an optical pupil position of the imaging unit for each captured image.   The image quality changing unit includes an illuminating unit that illuminates a photographic subject with auxiliary light for each captured image, and an illumination control unit that changes the illumination light amount of the illuminating unit for each captured image. The imaging device according to any one of claims 8 to 10.   The imaging apparatus according to claim 13, wherein the illuminating unit performs predetermined pattern projection on a photographic subject.   The discriminating unit discriminates the main subject region and the background region in the image based on a contrast evaluation value of an image formed on an image sensor by the photographing unit. 8. The imaging device according to any one of items 7.   The discriminating unit discriminates the main subject region and the background region in the image based on an imaging signal correlation value of an image formed in a different region on the image sensor by the photographing unit. The imaging device according to any one of claims 1 to 7. A shooting process for shooting a subject;
A determination step of determining a main subject region and a background region in the image shot by the shooting step;
A background image processing step of performing a blurring process on the background region determined in the determination step;
A composition change detection step for detecting a composition change in the photographing step;
A discriminating control step of operating the discriminating step and resetting the background region only when the composition change is detected by the composition change detecting step after operating the discriminating step at the start of moving image shooting and setting the background region; ,
And a moving image creating step of creating a moving image from a plurality of images subjected to the blurring process in the background image processing step.

Images