JP6097587B2 - Image reproducing apparatus and control method thereof - Google Patents

Description

  The present invention relates to an image reproducing apparatus capable of reconstructing a refocused image.

  In recent years, an imaging apparatus (light field camera) that can acquire not only the intensity distribution of light but also information on the incident direction of light in an imaging apparatus such as a digital camera has been proposed.

  For example, according to Non-Patent Document 1, a microlens array is arranged between a photographic lens and an image sensor, and one microlens is associated with a plurality of pixels of the image sensor, thereby allowing light that has passed through the microlens. Is acquired for each incident direction by a plurality of pixels. In addition to generating a normal captured image for the pixel signal (light field) acquired in this way, a technique called “Light Field Photography” is applied to any image plane (refocus plane). A focused image can be reconstructed after shooting.

  As a case where the user needs to reconstruct an image by refocusing, a case where the desired subject is not in focus when the image is confirmed after shooting can be considered. Also, there are cases in which an image focused on the subject in the foreground in some cases and the subject in the back in some cases is reconstructed and enjoyed. In these cases, it has been necessary to re-take an image with a conventional camera or to take a plurality of images in advance by shifting the focus position. However, a refocus image in which an arbitrary subject is focused can be reconstructed by shooting using a light field camera, and a desired image can be obtained without reshooting.

Ren.Ng and 7 others, “Light Field Photography with a Hand-Held Plenoptic Camera”, Stanford Tech Report CTSR 2005-02

  However, when reconstructing an image by refocusing, an operation for the user to determine a target to be focused on is necessary. In this regard, it is difficult to select a fine target with a limited operation member such as a digital camera, and it is cumbersome to determine a refocus target for each reproduction.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to reduce an operation load when a user determines a refocus target when playing back an image shot by a light field camera. is there.

An image reproduction apparatus according to the present invention includes an image data acquisition unit that acquires image data capable of generating a refocused image by combining pixel signals according to a focal position ;
A resulting unit preparative for obtaining distance information of the object in the image data, for each of a plurality of focal positions determined based on the distance information, a generating means for generating a refocus image using the image data, said generation Control means for causing the display means to display a plurality of refocus images generated by the means , wherein the control means is a subject based on a user operation in a state where the display means displays the plurality of refocus images. It is characterized by accepting the selection .

  ADVANTAGE OF THE INVENTION According to this invention, when reproducing | regenerating the image image | photographed with the light field camera, it becomes possible to reduce the operation load at the time of a user determining the refocusing object.

1 is a block diagram showing a configuration of an imaging apparatus that is a first embodiment of an image reproduction apparatus of the present invention. The figure explaining the structure of an image pick-up element and a micro lens array. 2A and 2B illustrate a configuration of a photographing lens, a microlens array, and an imaging element. The figure explaining the correspondence of the pupil region of a photographic lens, and a light reception pixel. The figure explaining the passage area of a refocus image generation light ray. 3 is a flowchart for explaining an image display operation of the image processing circuit according to the first embodiment of the present invention. The figure explaining the acquisition method of distance information in a 1st embodiment. The figure explaining the acquisition method of distance information in a 1st embodiment. FIG. 5 is a diagram illustrating a refocus plane selection method according to the first embodiment. FIG. 5 is a diagram illustrating a refocus plane selection method according to the first embodiment. The figure explaining the candidate image display method in 1st Embodiment. 9 is a flowchart for explaining an image display operation of an image processing circuit according to the second embodiment of the present invention. The figure explaining the acquisition method of distance information in a 2nd embodiment. FIG. 10 is a diagram for explaining a refocus plane selection method according to the second embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

(First embodiment)
FIG. 1 is a block diagram illustrating a configuration of an imaging apparatus 100 that is a first embodiment of an image reproduction apparatus according to the present invention. In FIG. 1, 101 is a photographing lens, 102 is a microlens array, and 103 is an image sensor. The light that has passed through the photographing lens 101 forms an image near the focal position of the photographing lens 101. The microlens array 102 includes a plurality of microlenses 1020, and is arranged in the vicinity of the focal position of the photographing lens 101 so that light that has passed through different pupil regions of the photographing lens 101 is divided and emitted for each pupil region. It has the function to do.

  The image sensor 103 is a photoelectric conversion element represented by a CMOS image sensor or a CCD image sensor. By arranging a plurality of pixels so as to correspond to one of the plurality of microlenses 1020, the light emitted by the microlens 1020 divided for each pupil region is received while maintaining the division information, and the data It has a function of converting into a processable image signal. As the imaging unit 1002, a configuration in which a plurality of cameras with different viewpoints are collectively regarded as the imaging unit 1002 as in Japanese Patent Application Laid-Open No. 2011-22796 is also effective in the present invention as an imaging optical system that can obtain light field data. The imaging optical system is not limited to the above as long as it can acquire light field data that is three-dimensional information of the subject space.

  Reference numeral 104 denotes an analog signal processing circuit (AFE), and 105 denotes a digital signal processing circuit (DFE). The analog signal processing circuit 104 performs correlated double sampling processing, signal amplification, reference level adjustment, A / D conversion processing, and the like on the image signal output from the image sensor 103. The digital signal processing circuit 105 performs digital image processing such as reference level adjustment on the image signal output from the analog signal processing circuit 104.

  Reference numeral 106 denotes an image processing circuit, 107 denotes a memory circuit, and 108 denotes a recording circuit. The image processing circuit 106 performs predetermined image processing and a refocusing operation, which is a feature of this embodiment, on the image signal output from the digital signal processing circuit 105. In addition, a face detection function is provided, and the face area of the subject can be detected from the image signal.

  The memory circuit 107 is a volatile memory that temporarily holds the image signal output from the image processing circuit 106 and stores calculation data. The recording circuit 108 is a recording medium such as a nonvolatile memory or a memory card for recording the image signal output from the image processing circuit 106.

  Reference numeral 109 denotes a control circuit, 110 denotes an operation circuit, and 111 denotes a display circuit. The control circuit 109 comprehensively drives and controls the entire imaging apparatus such as the imaging element 103 and the image processing circuit 106. The operation circuit 110 receives a signal from an operation member provided in the imaging apparatus 100 and reflects a user command to the control circuit 109. The display circuit 111 displays an image after shooting, a live view image, various setting screens, and the like.

  Next, the configuration of the photographic lens 101, the microlens array 102, and the imaging element 103 in the imaging apparatus of the present embodiment will be described.

  FIG. 2 is a diagram of the image sensor 103 and the microlens array 102 observed from the optical axis Z direction in FIG. One microlens 1020 is arranged so as to correspond to a plurality of unit pixels 201.

  A plurality of unit pixels 201 behind one microlens are collectively defined as a pixel array 20. In the present embodiment, the pixel array 20 has a total of 25 unit pixels 201 in 5 rows and 5 columns, and the image sensor 103 has 25 pixel arrays 20 in 5 rows and 5 columns.

  FIG. 3 is a diagram in which light emitted from the photographing lens 101 passes through one microlens 1020 and is received by the image sensor 103 as observed from a direction perpendicular to the optical axis Z. Light emitted from each of the pupil regions a1 to a5 of the photographing lens 101 and passed through the microlens 1020 forms an image on the corresponding rear unit pixels p1 to p5, respectively.

  FIG. 4A is a view of the aperture of the taking lens 101 as viewed from the optical axis Z direction. FIG. 4B is a view of one microlens 1020 and the pixel array 20 disposed behind the microlens 1020 when viewed from the optical axis Z direction. As shown in FIG. 4A, when the pupil region of the photographing lens 101 is divided into the same number of regions as pixels behind one microlens, one pixel is emitted from one pupil division region of the photographing lens 101. Light is imaged. However, here, it is assumed that the F-numbers of the photographing lens 101 and the microlens 1020 are substantially the same.

  The correspondence between the pupil division areas a11 to a55 of the photographing lens 101 shown in FIG. 4A and the pixels p11 to p55 shown in FIG. 4B is point-symmetric when viewed from the optical axis Z direction. Therefore, the light emitted from the pupil division region a11 of the photographing lens 101 forms an image on the pixel p11 in the pixel array 20 behind the microlens. Similarly, the light emitted from the pupil division area a11 and passing through another microlens 1020 forms an image on the pixel p11 in the pixel array 20 behind the microlens.

  Here, a method for calculating a focal position (refocus plane) corresponding to an arbitrary subject position in the screen (in the image) will be described.

  As described with reference to FIG. 4, each pixel of the pixel array 20 receives light that has passed through different pupil regions with respect to the photographing lens 101. By combining a plurality of pixel signals from these divided signals, a pair of signals divided into pupils in the horizontal direction is generated.

  Expression (1) is obtained by integrating the light passing through the left area (pupil areas a11 to a51, a12 to a52) of the exit pupil of the photographing lens 101 for each pixel of a certain pixel array 20. This is applied to a plurality of pixel arrays 20 arranged in the horizontal direction, and a subject image constituted by these output signal groups is defined as an A image. Expression (2) is obtained by integrating the light passing through the right area (pupil areas a14 to a54 and a15 to a55) of the exit pupil of the photographing lens 101 for each pixel of a certain pixel array 20. This is applied to a plurality of pixel arrays 20 arranged in the horizontal direction, and a subject image constituted by these output signal groups is defined as a B image. A correlation operation is performed on the A image and the B image, and an image shift amount (pupil division phase difference) is detected. Furthermore, the focal position corresponding to an arbitrary subject position in the screen can be calculated by multiplying the image shift amount by the focal position of the photographing lens 101 and a conversion coefficient determined by the optical system.

  Next, an image is reconstructed at an arbitrarily set focal position (refocus plane) with respect to imaging data (image data) acquired by the configuration of the photographing lens 101, the microlens array 102, and the imaging element 103 described above. Processing (processing for enabling refocusing) will be described.

  FIG. 5 shows from which the light passing through a certain pixel on the refocus plane set arbitrarily is emitted from which pupil division region of the photographing lens and which microlens is incident from a direction perpendicular to the optical axis Z. FIG. In FIG. 5, the position of the pupil division area of the photographing lens is the coordinates (u, v), the pixel position on the refocus plane is the coordinates (x, y), and the position of the microlens on the microlens array is the coordinates (x ′, y ′), the distance from the taking lens to the microlens array is F, and the distance from the taking lens to the refocus plane is αF. α is a refocus coefficient for determining the position of the refocus plane and can be arbitrarily set by the user. In FIG. 5, only the directions of u, x, and x ′ are shown, and v, y, and y ′ are omitted. As shown in FIG. 5, the light passing through the coordinates (u, v) and the coordinates (x, y) reaches the coordinates (x ′, y ′) on the microlens array. The coordinates (x ′, y ′) can be expressed as in Equation (3).

When the output of the pixel receiving this light is L (x ′, y ′, u, v), the output E (x, y) obtained with the coordinates (x, y) on the refocus plane is L Since (x ′, y ′, u, v) is integrated with respect to the pupil region of the photographing lens, Equation (4) is obtained.

  In Expression (4), since the refocus coefficient α is determined by the user, if (x, y) and (u, v) are given, the position (x ′, y ′) of the microlens where the light enters can be known. . Then, the pixel corresponding to the position (u, v) is known from the pixel array 20 corresponding to the microlens. The output of this pixel is L (x ', y', u, v). E (x, y) can be calculated by performing this operation for all pupil division regions and integrating the obtained pixel outputs. If (u, v) is the representative coordinates of the pupil division area of the taking lens, the integral of equation (4) can be calculated by simple addition.

  Next, the image display operation of the present embodiment will be described with reference to FIGS. 6, 7, 8, 9, 10, and 11.

  FIG. 6 is a flowchart showing an image display operation in the present embodiment. In FIG. 6, in step S <b> 601, image data obtained from the image sensor 103 is acquired.

  In step S602, the input imaging data is divided into predetermined distance information acquisition areas (hereinafter referred to as “D area”). Here, since the number of divisions of the D area (divided region) is reflected in the focusing accuracy at the time of reconstruction, it is desirable that the division is performed for each pixel array 20 having the minimum position resolution. However, the number of pixel arrays 20 can be determined as appropriate according to restrictions on the calculation capability, frame rate, and the like of the image processing circuit 106, such as forming a single D area. That is, the division number of the D area may be decreased as the frame rate is higher or the calculation capability of the image processing circuit 106 is lower. In this embodiment, the configuration of the D area is a 5 × 5 pixel array 20.

  In step S603, the above-described formulas (1) and (2) are applied to the imaging data divided into the D areas, and the subject distance representing each D area is calculated. Here, the result of only the central pixel array 20 may be substituted, or the average value of the 5 × 5 pixel array 20 may be used. The distance information of the subject for each D area obtained in this way is defined as a distance information map.

  In step S604, a refocus plane is selected according to the above-described distance information map. At this time, the refocus plane to be selected is a method of acquiring a mode value from a frequency distribution of a distance information map (hereinafter referred to as a distance information histogram), a method of applying curve fitting by a predetermined function form, etc. Selected by.

  In the present embodiment, an example in which a refocus plane is selected for each predetermined distance area is shown. Here, the distance area is classified into, for example, a short distance (up to several tens of centimeters), a middle distance (up to several tens of meters), and a long distance (up to several tens of meters to infinity). Refers to an area. The short distance, medium distance, and long distance ranges may be set in a fixed range regardless of the image and the shooting conditions, or may be set equally, for example, based on the refocusable range. The refocus range is obtained as follows. Assuming that the angle resolution Δθ, the number of angle divisions Nθ, and the pixel pitch Δx, the maximum refocus amount dmax is calculated by the following equation (5).

dmax = Nθ · Δx / tan (Δθ) (5)
That is, the range of the distance dmax before and after the optical axis direction around the focused distance when the refocusable data is captured is the refocusable range, and the subject within the range can be focused.

  In step S605, the above-described reconstruction process is performed on each refocus plane to generate a refocus image (refocus image generation). In step S606, the number of refocus planes selected in step S604 is determined. If a plurality of refocus surfaces are selected, the process proceeds to step S607. If a single surface is selected, the process proceeds to step S609. In step S <b> 607, a plurality of refocus images generated in step S <b> 605 are displayed on the display circuit 111.

  In step S608, it is determined whether or not the user has selected any of the displayed refocus images. If it is selected, the process returns to step S609. If it is not selected, the process returns to step S608 to wait for a selection operation. In step S609, a refocus image on a certain refocus plane is displayed on the display circuit 111.

  Next, the distance information acquisition method in steps S602 and S603 will be described with reference to two cases of FIGS. FIG. 7A shows acquired image data. When an image is taken, there are three types of subjects, such as a subject person 701 at a short distance, a subject person 702 at a medium distance, and a subject building 703 at a long distance. An example in which a subject exists in the distance area is shown. In this imaging data, the subject distance is calculated for each D area. FIG. 7B is a schematic diagram expressing these subject distances as a contrast diagram.

  Further, FIG. 8A shows acquired image data, and shows an example in which a subject person 801 exists only at a short distance. In this imaging data, the subject distance is calculated for each D area. FIG. 8B is a schematic diagram expressing these subject distances as a contrast diagram.

  Next, the refocus plane selection method in step S604 will be described. 9 and 10 are schematic diagrams of distance information histograms in which the subject distance information for each D area obtained in step S603 is represented by the subject distance on the horizontal axis and the frequency on the vertical axis. 9 is a schematic diagram of the distance information histogram of the image of FIG. 7, and FIG. 10 is a schematic diagram of the distance information histogram of the image of FIG. Among these, a part having a frequency equal to or higher than a certain threshold in each distance area is selected as a refocus plane.

  FIG. 9 shows an example in which locations (refocus plane (a), refocus plane (b), and refocus plane (c)) exceeding the threshold value appear in three distance areas, short distance, medium distance, and long distance. ing. In this case, three refocus planes are selected.

  FIG. 10 shows an example in which a location (refocus plane (a)) exceeding the threshold appears in a short distance area. In this case, one refocus plane is selected. Here, an example is shown in which the mode value is selected at a location that exceeds the threshold value in each of the three distance regions of short distance, medium distance, and long distance. However, the present invention is not limited to this. The number of can be specified by an arbitrary value.

  In general, it is reasonable to consider that the mode value of the distance information histogram corresponds to the position where a representative subject exists. By detecting this mode value, it is possible to select a refocus plane on which a representative subject exists. The detection method is not limited to the mode value, and a known method such as curve fitting using a predetermined function form can be applied.

  Next, the image display operation in steps S607, S608, and S609 will be described. FIG. 11 shows an example in which a refocus image is presented to the user on the display circuit 111. FIG. 11A shows an example in which refocus images reconstructed on the refocus plane (a), the refocus plane (b), and the refocus plane (c) in FIG. 9 are displayed. Here, the user can confirm a plurality of refocus images and can select an image he / she wants to see. As for the operation after selection, for example, the operation of the display circuit 111 may be switched so as to display only the selected image as shown in FIG. 11B so that the user can easily confirm details. You may display larger than another image so that the selected image may be emphasized like c). Further, the imaging data and the display operation performed in step S609 may be stored in association with each other, and the same display operation may be performed without user operation when the same imaging data is reproduced next time.

  Although not shown in the flowchart of FIG. 6, when it is determined in step S606 that there are a plurality of refocus planes, the following may be performed. That is, after the operation of displaying the image selected in step S609, the process returns to step S607 based on the user's instruction, and a plurality of refocus images are displayed so that the image can be selected again.

  As described above, it is possible to reduce the user's operation load by determining the refocus plane from the distance information of the image and presenting the candidate of the refocus image to the user.

(Second Embodiment)
Hereinafter, the second embodiment of the present invention will be described with reference to FIGS. 12, 13, and 14. The second embodiment differs from the first embodiment in that subject information is taken into account when determining refocus image candidates. In addition, since the structure of the imaging device 100 in 2nd Embodiment is the same as that of 1st Embodiment, the description is abbreviate | omitted using the same description code | symbol.

  Next, the image display operation of this embodiment will be described with reference to the flowchart of FIG. The operations from step S1201 to S1206 are the same as the operations from step S601 to S606 in FIG. 6 of the first embodiment, and a description thereof will be omitted.

  Next, the distance information acquisition method in steps S1202 and S1203 will be described with reference to FIG. FIG. 13A shows acquired image data, and shows an example in which a subject exists in three distance areas, a short distance, a medium distance, and a long distance when an image is captured. In this imaging data, the subject distance is calculated for each D area. FIG. 13B is a schematic diagram expressing these subject distances as a contrast diagram. The short distance includes subjects of the person 1301 and the plant 1302. Further, the vehicle 1303 is included in the middle distance, and the subject of the building 1304 is included in the long distance.

  Next, a method for selecting a refocus plane in step S1204 will be described. FIG. 14 is a schematic diagram of a distance information histogram in which subject distance information for each D area obtained in step S1203 is represented with the horizontal axis representing subject distance and the vertical axis representing frequency. Among these, a part having a frequency equal to or higher than a certain threshold is selected as a refocus plane.

  In FIG. 14, when an image as shown in FIG. 13 is taken, a location (refocus surface (e), refocus surface (f), refocus surface (f), An example in which the refocus plane (g) and the refocus plane (h)) appear is shown. In this case, four refocus planes are selected.

  In step S1207, it is determined whether or not there are a plurality of refocus surfaces within the same distance range for the refocus surface selected in step S1204. In the example of FIG. 14, since there are a plurality of refocus surfaces (refocus surface (e) and refocus surface (f)) at a short distance, the process proceeds to step S1208. If there are no refocus planes in the same distance range, the process advances to step S1211.

  In step S1208, it is determined whether subject information is included in the refocus image for the refocus image within the same distance range. Here, it is assumed that a human face is detected as subject information. For example, the refocus image reconstructed on the refocus plane (e) is focused on the person, and the refocus image reconstructed on the refocus plane (f) is focused on the plant. Suppose that there is a match. In this case, when face detection is performed by the image processing device 106, it can be determined that the refocus image reconstructed on the refocus plane (e) is an image including subject information. In this case, the process proceeds to step S1209, and the refocus plane (e) including the subject information is selected as the short-distance refocus plane. If the subject information is not included, the process proceeds to step S1210, and the refocus plane showing the mode value in the distance information histogram is selected as the short distance refocus plane. In addition, when a plurality of refocus images within the same distance range include a person's face, priority may be given to a refocus image in which the face is displayed with a larger percentage of the area. Alternatively, a known person's face and another person's face may be distinguished by an arbitrary method, and a refocus image including the known person may be prioritized.

  In steps S1211 to S1213, an image display operation is performed in the same manner as in steps S607 to S609 of the first embodiment. In the image display in step S1211, priority processing such as displaying a refocus image including subject information in a larger area than other refocus images may be performed so that the user can easily determine the image of the subject. Good.

  In the present embodiment, when there are a plurality of refocus image candidates in a certain distance area, an example in which a refocus image including subject information is preferentially displayed is shown. However, the present invention is not limited to this, and the refocus image including the subject information may be given priority over the refocus image of all the distance regions, and for example, only the refocus image including the subject information may be presented to the user as a candidate. .

  As described above, by determining the refocus plane in consideration of subject detection information in addition to the image distance information, a more suitable refocus image candidate can be presented to the user, and the user's operation load Can be reduced.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

Claims (15)

Image data acquisition means for acquiring image data capable of generating a refocus image by synthesizing pixel signals according to the focal position ;
A resulting unit preparative for obtaining distance information of the object in the image data,
Generating means for generating a refocus image using the image data for each of a plurality of focal positions determined based on the distance information;
Control means for causing a display means to display a plurality of refocus images generated by the generating means ,
The image reproduction apparatus according to claim 1, wherein the control means receives selection of a subject based on a user operation in a state where the plurality of refocus images are displayed on the display means . The image reproducing apparatus according to claim 1, wherein the generation unit sets the focal position for each predetermined range in distance information .   The image reproducing apparatus according to claim 2, further comprising setting means for setting the predetermined range.   The image reproducing apparatus according to claim 3, wherein the generation unit sets the predetermined range based on a refocusable range of the image data. The generation unit, based on the distance information, and generates the refocus image position ratio in the image data corresponding to the distance information of a larger object than a predetermined value as the focal position The image reproduction device according to claim 1. 6. The generating means generates a histogram of the distance information corresponding to each region of the image data, and determines a focus position for generating the refocused image based on the histogram. The image reproducing device according to any one of the above. Detecting means for detecting subject information in the image;
Discriminating means for discriminating whether or not the subject information is included in the refocus image,
7. The control unit according to claim 1, wherein the display unit preferentially displays a refocus image determined by the determination unit to include the subject information. The image reproducing device described. The image reproduction apparatus according to claim 1, wherein the control unit emphasizes a refocus image selected by a user operation and displays the refocus image on the display unit. The control unit is configured to display the refocus image on the display unit when the generation unit generates one refocus image based on the distance information and when there are a plurality of focus positions. The image reproducing device according to claim 1, wherein the image reproducing device is switched.   The image reproducing apparatus according to claim 1, wherein the image data is a plurality of parallax image data having parallax with each other.   The image reproduction apparatus according to claim 1, wherein the image data is image data captured by a light field camera.   12. The image reproducing apparatus according to claim 1, wherein the image data is image data obtained from an imaging device in which a plurality of photoelectric conversion elements are associated with one microlens. . The image reproduction device according to any one of claims 1 to 12, wherein the distance information is information on a subject distance indicating a distance from an imaging device that has captured the image data to a subject. The said generation means produces | generates the said refocus image by synthesize | combining the pixel signal of the said image data corresponding to each pixel in the focus position of the object of refocus, The one of Claims 1 thru | or 13 characterized by the above-mentioned. The image reproducing device according to item. An image data acquisition step of acquiring image data capable of generating a refocused image by combining pixel signals according to the focal position ;
A resulting step preparative for obtaining distance information of the object in the image data,
For each of a plurality of focal positions determined based on the distance information, a generation step of generating a refocus image using the image data ;
And a control step of displaying on the display means a plurality of refocus images generated at the generating step,
In the control step, a selection of a subject based on a user operation is received in a state where the plurality of refocus images are displayed on the display means .

Images