JP6168549B2 - Image processing apparatus, image processing method, program, and imaging apparatus - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, a program, and an imaging apparatus, and more particularly to a technique for obtaining an optimal image by combining a plurality of images.

  In recent years, digital cameras, mobile phone terminals with a photographing function, tablet terminals, and the like have become widespread, and photographing using these cameras is also routinely performed. However, it may be difficult to obtain a photographed image that matches the photographer's intention in one photographing. In particular, when there are a large number of people as subjects, such as when taking a group photo, someone closes their eyes at the moment of shooting, so that all the subjects have a good expression on a single photo. Is difficult.

  For this reason, by continuously shooting a plurality of subjects and combining a plurality of photographed images obtained by continuous shooting, a composite image in which all the subjects have a good expression is generated. For example, in Patent Document 1, a facial image area having the highest facial expression evaluation value is extracted from a plurality of photographed images, and the extracted facial image area is inserted into a background image from which the facial image area has been removed in advance. A technique for generating a composite image in which all subjects have a good expression is described.

JP 2007-299297 A

  By the way, as described in Patent Document 1, when a face image region extracted from another photographed image is to be inserted into a background image from which the face image region has been removed, a space between the inserted face image region and the background image is used. Therefore, it is possible that an image shift occurs. In such a case, the generated composite image is unnatural because the face image area and the background image are not matched.

  Patent Document 1 also describes that when a misalignment occurs between the embedded face image area and the background image, the misalignment is eliminated by adjusting the synthesis position and size of the face image area. Has been. However, if such adjustment is repeatedly performed until the positional deviation is eliminated, it takes a long time to complete the composite image. Even if the positional deviation between the face image area and the background image can be eliminated, the boundary between the inserted face image area and the background image may be recognized as a line. In such a case, the technique described in Patent Literature 1 cannot eliminate unnaturalness in the composite image.

  The present invention has been made in view of such a point, and an object thereof is to easily generate a more natural composite image using a plurality of photographed images.

The image processing apparatus according to the present invention includes a captured image selection unit, a multi-resolution decomposition unit, a replacement processing unit, and a composite image generation unit. The configuration and function of each unit are as follows. The captured image selection unit selects a replacement captured image on the side where the image is to be replaced and a replacement-side captured image on the side where the image is to be replaced from among a plurality of continuously captured images. The multi-resolution decomposition unit includes a first area including at least a replacement area outer peripheral area that is an area around the outer periphery of the replacement area in the replacement-side captured image, and an area around the outer periphery of the replacement area in the replacement captured image. A second region including at least a peripheral region around a certain replacement region is decomposed into multiple resolutions. The replacement processing unit replaces at least a part of the second region with at least a part of the first region in each frequency component obtained by the decomposition by the multi-resolution decomposition unit. Composite image generation unit, first processing for reconstructing the respective frequency components replacement is performed by the replacement processor, or combines the image replacement area as the reconstructed image obtained in the first processing A composite image is generated by the second process. Then, when the first area is the replacement area outer peripheral area, the multi-resolution decomposition unit performs the replacement area outer peripheral area as the first area and the replacement area outer peripheral area as the second area, respectively. The permutation processing unit decomposes into multiple resolutions, and the permutation processing unit includes the perimeter region around the replacement region in each frequency component around the perimeter region of the permutation region and each perimeter component around the permutation region obtained by the multiresolution resolution unit. The inner peripheral side region is replaced with the inner peripheral side region of the replacement region outer peripheral peripheral region, and the composite image generation unit generates a composite image by the second processing. When the first area is the entire replacement-side captured image, the multi-resolution decomposition unit multi-resolutions the entire replacement-side captured image as the first area and the entire replacement-captured image as the second area. The replacement processing unit replaces the replacement region with a replacement region in each frequency component of the entire replacement-side captured image and each frequency component of the entire replacement captured image obtained by the multi-resolution decomposition unit. The composite image generation unit generates a composite image by the first process.

The image processing method of the present invention includes a captured image selection step, a multi-resolution decomposition step, a replacement processing step, and a composite image generation step. First, in a photographed image selection step, a replacement photographed image on the side to be replaced and a replacement-side photographed image on the image replacement side are selected from a plurality of continuously photographed images. . Subsequently, in the multi-resolution decomposition step, a first area including at least a replacement area outer peripheral area that is an area around the outer periphery of the replacement area in the replacement-side captured image, and an outer periphery of the replacement area in the replacement captured image And the second area including at least the peripheral area of the outer periphery of the replacement area, which is an area of the area, is decomposed into multiple resolutions. Subsequently, in each frequency component obtained by decomposing into multiple resolutions in the replacement processing step, at least a part of the second region is replaced with at least a part of the first region. Subsequently, a synthetic image generating step, the first processing for reconstructing the respective frequency components is performed replaced, or, the synthesizing the image replacement area as the reconstructed image obtained in the first processing A composite image is generated by the process 2. When the first area is a replacement area outer peripheral area, the multi-resolution decomposition step includes a replacement area outer peripheral area as the first area and a replacement area outer peripheral area as the second area, respectively. In the replacement processing step, each frequency component in the peripheral area around the replacement area and each frequency component in the peripheral area around the replacement area obtained in the replacement processing step are included in the peripheral area around the replacement area. The peripheral side region is replaced with the inner peripheral side region of the replacement region outer peripheral peripheral region, and in the composite image generation step, a composite image is generated by the second process. When the first area is the entire replacement-side captured image, the multi-resolution decomposition step includes the entire resolution-side captured image as the first area and the entire replacement-captured image as the second area. In the replacement processing step, the replacement area is replaced with a replacement area in each frequency component of the entire replacement-side captured image obtained by decomposing into multiple resolutions and each frequency component of the entire replacement captured image. In the image generation step, a composite image is generated by the first process.

In addition, the program of the present invention causes a computer to realize each process of a captured image selection step, a multi-resolution decomposition step, a replacement processing step, and a composite image generation step . First, in a photographed image selection step, a replacement photographed image on the side to be replaced and a replacement-side photographed image on the image replacement side are selected from a plurality of continuously photographed images. . Subsequently, in the multi-resolution decomposition step, a first area including at least a replacement area outer peripheral area that is an area around the outer periphery of the replacement area in the replacement-side captured image, and an outer periphery of the replacement area in the replacement captured image And the second area including at least the peripheral area of the outer periphery of the replacement area, which is an area of the area, is decomposed into multiple resolutions. Subsequently, in each frequency component obtained by decomposing into multiple resolutions in the replacement processing step, at least a part of the second region is replaced with at least a part of the first region. Subsequently, a synthetic image generating step, the first processing for reconstructing the respective frequency components is performed replaced, or, the synthesizing the image replacement area as the reconstructed image obtained in the first processing A composite image is generated by the process 2. When the first area is a replacement area outer peripheral area, the multi-resolution decomposition step includes a replacement area outer peripheral area as the first area and a replacement area outer peripheral area as the second area, respectively. In the replacement processing step, each frequency component in the peripheral area around the replacement area and each frequency component in the peripheral area around the replacement area obtained in the replacement processing step are included in the peripheral area around the replacement area. The peripheral side region is replaced with the inner peripheral side region of the replacement region outer peripheral peripheral region, and in the composite image generation step, a composite image is generated by the second process. When the first area is the entire replacement-side captured image, the multi-resolution decomposition step includes the entire resolution-side captured image as the first area and the entire replacement-captured image as the second area. In the replacement processing step, the replacement area is replaced with a replacement area in each frequency component of the entire replacement-side captured image obtained by decomposing into multiple resolutions and each frequency component of the entire replacement captured image. In the image generation step, a composite image is generated by the first process.

The imaging apparatus of the present invention includes an imaging unit, a captured image selection unit, a multi-resolution decomposition unit, a replacement processing unit, a composite image generation unit, and a storage unit, and the configuration and function of each unit. Do as follows. The imaging unit continuously captures a plurality of captured images. The captured image selection unit selects a replacement captured image on the side where the image is to be replaced and a replacement-side captured image on the side where the image is to be replaced from among a plurality of continuously captured images. The multi-resolution decomposition unit includes a first area including at least a replacement area outer peripheral area that is an area around the outer periphery of the replacement area in the replacement-side captured image, and an area around the outer periphery of the replacement area in the replacement captured image. A second region including at least a peripheral region around a certain replacement region is decomposed into multiple resolutions. The replacement processing unit replaces at least a part of the second region with at least a part of the first region in each frequency component obtained by the decomposition by the multi-resolution decomposition unit. Composite image generation unit, first processing for reconstructing the respective frequency components replacement is performed by the replacement processor, or combines the image replacement area as the reconstructed image obtained in the first processing A composite image is generated by the second process. The storage unit stores the composite image generated by the composite image generation unit. Then, when the first area is the replacement area outer peripheral area, the multi-resolution decomposition unit performs the replacement area outer peripheral area as the first area and the replacement area outer peripheral area as the second area, respectively. The permutation processing unit decomposes into multiple resolutions, and the permutation processing unit includes the perimeter region around the replacement region in each frequency component around the perimeter region of the permutation region and each perimeter component around the permutation region obtained by the multiresolution resolution unit. The inner peripheral side region is replaced with the inner peripheral side region of the replacement region outer peripheral peripheral region, and the composite image generation unit generates a composite image by the second processing. When the first area is the entire replacement-side captured image, the multi-resolution decomposition unit multi-resolutions the entire replacement-side captured image as the first area and the entire replacement-captured image as the second area. The replacement processing unit replaces the replacement region with a replacement region in each frequency component of the entire replacement-side captured image and each frequency component of the entire replacement captured image obtained by the multi-resolution decomposition unit. The composite image generation unit generates a composite image by the first process.

By configuring the image processing apparatus or the imaging apparatus as described above and performing image processing, the first area including at least the replacement area outer peripheral area and the second area including at least the replacement area outer peripheral area are provided. Are synthesized under each frequency domain obtained by converting to multi-resolution. Then, the reconstructed image obtained by reconstructing after synthesis, or by using the images of the reconstructed image and substitutions regions, the composite image is generated. At this time, if the first area is a perimeter area of the replacement area, the perimeter area of the replacement area as the first area and the perimeter area of the replacement area as the second area have multiple resolutions, respectively. In each frequency component of the perimeter area of the replacement area and each frequency component of the perimeter area of the replacement area that has been decomposed into multiple resolutions, the inner peripheral area of the perimeter area of the replacement area is within the perimeter area of the replacement area. Replaced with the circumferential area. Then, a synthesized image is generated by synthesizing each frequency component subjected to the replacement and the image in the replacement area. When the first area is the entire replacement-side captured image, the entire replacement-side captured image as the first area and the entire replacement-captured image as the second area are decomposed into multiple resolutions. The replacement area is replaced with the replacement area in each frequency component of the entire replacement-side captured image decomposed into multiple resolutions and each frequency component of the entire replacement captured image. Then, a composite image is generated by reconstructing each frequency component subjected to the replacement. As a result, the boundary portion between the replacement portion and the replacement portion becomes inconspicuous at the place where the image replacement has been performed.

  According to the present invention, a more natural composite image can be easily generated using a plurality of photographed images.

It is a schematic diagram explaining the outline | summary of this invention, and is a figure which shows the example of the picked-up image containing a good expression and a bad expression of a person. It is a schematic diagram explaining the outline | summary of this invention, and is a figure which shows the example of a production | generation of the synthesized image as a best shot. 1 is a block diagram illustrating a configuration example of an image processing apparatus according to an embodiment of the present invention. It is a figure which shows the example of the extraction process of the black eye area | region which concerns on one embodiment of this invention. It is a figure which shows the example of a determination of the eye opening degree which concerns on the example of 1 embodiment of this invention. It is a figure which shows the acquisition example of the pixel of the lip line which concerns on one embodiment of this invention. It is a figure which shows the example of determination of the raising degree of a mouth angle which concerns on the example of 1 embodiment of this invention. It is a figure which shows the example of a setting of the ellipse for replacement area | region extraction based on one embodiment of this invention. It is a figure which shows the structural example of the multi-resolution decomposition part which concerns on the example of 1 embodiment of this invention. It is a figure which shows the example of the decomposition | disassembly process of the multi-resolution decomposition part which concerns on one embodiment of this invention. It is a figure which shows the example of the replacement process of the image which concerns on the 1st Example of this invention. It is a figure which shows the example of the replacement process of the image in each frequency component based on the 1st Example of this invention. It is a figure which shows the structural example of the reconstruction part which concerns on the example of 1 embodiment of this invention. It is a flowchart which shows the example of the image processing method which concerns on the 1st Example of this invention. FIG. 15 is a diagram showing a comparison between a composite image according to the first exemplary embodiment of the present invention and a composite image generated by a conventional technique, FIG. 15A is a composite image by a conventional technique, and FIG. 15B is a first implementation. It is the synthesized image produced | generated by the form example. It is a figure which shows the example of a setting of the area | region which performs decomposition | disassembly to multi-resolution based on the 2nd Example of this invention. It is a figure which shows the example of the position and range of the area | region which performs decomposition | disassembly to the multi-resolution based on the 2nd Example of this invention. It is a figure which shows the example of the replacement process of the image which concerns on the 2nd Example of this invention. It is a figure which shows the structural example of the synthesized image which concerns on the 2nd Example of this invention. It is a block diagram which shows the structural example of the image processing apparatus which concerns on the 2nd Example of this invention. It is a figure explaining the pixel area | region required for the discrete wavelet transformation process which concerns on the 2nd example of this invention, the replacement and composition process of an image, and an inverse wavelet transformation process. It is a flowchart which shows the example of the image processing method which concerns on the 2nd Example of this invention. 1 is a block diagram illustrating a configuration example of an imaging apparatus according to an embodiment of the present invention.

An example of an image processing apparatus and an image processing method according to an embodiment of the present invention will be described in the following order with reference to the drawings. However, the present invention is not limited to the following examples.
1. 1. Outline of the present invention First Embodiment Example 3 Second Embodiment Example 4 4. Configuration example of imaging apparatus according to an embodiment of the present invention Various modifications

[1. Outline of the present invention]
First, the outline | summary of this invention is demonstrated with reference to FIG.1 and FIG.2. FIG. 1 is a diagram illustrating three captured images Po1, a captured image Po2, and a captured image Po3 captured continuously in time by an imaging device (not shown). The photographed image Po1 to the photographed image Po3 show a person Ps1, a person Ps2, and a person Ps3 as subjects. In the photographed image Po1, the eyes of the central person Ps2 are closed, and in the photographed image Po2, the eyes of the left end, the person Ps2, and the right end Ps3 are closed. In the photographed image Po3, the eyes of the rightmost person Ps3 are closed.

  On the other hand, these three photographed images Po1 to Po3 also show good expressions of the persons Ps1, Ps2, and Ps3. The rightmost person Ps3 is smiling in the captured image Po1, the central person Ps2 is smiling in the captured image Po2, and the leftmost person Ps1 is smiling in the captured image Po3.

  Therefore, in each of the persons Ps1 to Ps3, by replacing the image of the face area having a good expression from the photographed image Po1 to the photographed image Po3 with the image of the face area having no good expression, the good expression of all the subjects. It is thought that a photograph (best shot) containing However, if the photographed image Po on the side where the image is replaced appears to be different depending on the person Ps, a plurality of composite images are simply generated. For this reason, one of the photographed images Po1 to Po3 is selected as a replacement photographed image that is the basis of the composite image as the best shot.

  In the example shown in FIG. 1, the middle-stage photographed image Po2 is selected as the replacement photographed image. Since the person Ps2 shown in the center of the photographed image Po2 has a good expression, the image of the face region Rf22 is used as it is. For the right person Ps3, the image in the face area Rf31 extracted from the photographed image Po1 in the top row is replaced with the image of the face area in which the eyes are closed. For the left person Ps1, the image in the face area Rf13 extracted from the lowermost photographed image Po3 is replaced with the image of the face area in which the eyes are closed.

  By performing such processing, as shown in FIG. 2, a composite image Pc in which all of the persons Ps1 to Ps3 have a good expression is generated as the best shot. In the present invention, by performing image replacement under the discrete wavelet transform region, it is possible to generate a natural composite image in which the boundary between the replacement portion and the replacement portion is not conspicuous.

[2. First Embodiment]
(2-1. Configuration Example of Image Processing Device)
FIG. 3 shows an internal configuration example of the image processing apparatus 100 according to the first embodiment of the present invention. The image processing apparatus 100 generates a composite image Pc as a best shot by using a plurality of photographed images Po continuously photographed (continuously shot) by the imaging apparatus 200 as an input. The image processing apparatus 100 includes a face detection unit 110, a captured image selection unit 120, and a composite image generation unit 130.

  The face detection unit 110 includes a face part detection unit 111 and an eye / mouth part detection unit 112. The face portion detection unit 111 detects a face portion of a person from each of a plurality of captured images Po input from the imaging device 200. The eye / mouth part detection unit 112 detects the eye and mouth parts from the face of the person detected by the face part detection unit 111.

  The detection of the face part by the face part detection unit 111 and the detection of the eyes and mouth part by the eye / mouth part detection unit 112 are performed by using a commonly used face area detection function or a face part detection function. be able to. In the present embodiment example, it is assumed that object detection using a Haar-like feature quantity and a classifier having a Cascade structure prepared as a function in OpenCV is used. OpenCV is an open source computer vision library. The object detection using the discriminator described above is to detect parts such as the face, eyes, and mouth by looking at the brightness of the target area.

  The captured image selection unit 120 includes a facial expression estimation unit 121 and a replacement / replacement captured image selection unit 122. The facial expression estimation unit 121 estimates the facial expression of a person by analyzing the images of the eyes and mouth detected by the eye / mouth part detection unit 112. Based on the facial expression information estimated by the facial expression estimation unit 121, the replacement side / replacement captured image selection unit 122 selects the replacement side captured image including a good facial expression image from the plurality of captured images Po, and the best A replacement photographed image as a base of the shot is selected. Details of the processing performed by the captured image selection unit 120 will be described later with reference to FIGS.

  The composite image generation unit 130 includes a replacement area range determination unit 131, a multi-resolution decomposition unit 132, a replacement processing unit 133, and a reconstruction unit 134. The replacement area range determination unit 131 detects a face portion having the best facial expression from the replacement side captured image selected by the replacement side / replacement captured image selection unit 122. Then, a range to be cut out as a replacement area is calculated using parameters obtained by analyzing the detected face image. The replacement area is cut out by the subsequent multi-resolution decomposition unit 132, and only the range is determined here.

  In the present embodiment, an area that includes the entire face of the person and is slightly larger than the entire face is cut out with an ellipse as the replacement area. Therefore, the replacement region range determination unit 131 calculates the major axis and minor axis of this ellipse. The major axis and minor axis of the ellipse can be calculated using, for example, the distance between the eyes detected by the eye / mouth part detection unit 112. Details of the method of calculating the major axis and minor axis of the ellipse will be described later.

  Since the size of the ellipse is considered to be different for each person Ps shown as a subject, the replacement area determination processing is performed for each person Ps that needs to replace the face area. In the example shown in FIG. 1, for the person Ps3, the replacement area range determination unit 131 extracts the size of the face area Rf31 extracted as a replacement area from the face image in the captured image Po1, that is, the size of the ellipse. To decide. For the person Ps1, the size of the face area Rf13 to be extracted as a replacement area is determined from the face image in the captured image Po3.

  The multi-resolution decomposition unit 132 decomposes the replacement-captured image and the replacement-side captured image selected by the replacement-side / replacement-captured image selection unit 122 into multiple resolutions by discrete wavelet transform. The replacement processing unit 133 uses the ellipse whose size is determined by the replacement region range determination unit 131 to extract a face region as an update region from each frequency component decomposed into multiple resolutions. Then, in each frequency component, a process is performed to replace a face area including a face with a good expression with a face area including a face with a good expression. The reconstruction unit 134 reconstructs an image by performing inverse wavelet transform on each frequency component subjected to face region replacement. Thereby, the composite image Pc as the best shot is generated. Details of processing in each of these units constituting the composite image generation unit 130 will be described later with reference to FIGS. 8 to 13.

<2-1-1. Details of processing of photographed image selection unit>
Next, details of the processing of the captured image selection unit 120 will be described. First, details of the processing of the facial expression estimation unit 121 will be described with reference to FIGS. The facial expression estimation unit 121 estimates the facial expression of the person Ps in the captured image Po by performing image analysis on the images of the eye part and the mouth part detected by the eye / mouth part detection unit 112. In the present embodiment, an expression satisfying the following two points is regarded as a good expression (smile).
1. 1. The eyes are open to some extent. Threshold angle processing is applied to the one eye area detected by the eye / mouth part detection unit 112 for the degree of opening of the eye whose mouth corner is the highest among the faces of the person Ps shown in the plurality of photographed images Po. Then, the black eye part is detected, and the determination is made based on the size of the detected black eye part. The degree of mouth angle rise is estimated by approximating the lip line with a quadratic function and the magnitude of the coefficient of the quadratic term.

  First, a method for estimating the degree of eye opening will be described with reference to FIG. FIG. 4 is a diagram illustrating an example of threshold processing applied to the one-eye region. The left side of FIG. 4 shows the one-eye region detected by the eye / mouth part detection unit 112, and the result of applying threshold processing to this region is shown on the right side of FIG. On the right side of FIG. 4, the black eye portion detected as a result of the threshold processing is displayed as a white pixel, and the degree of opening of the eye is determined based on the number of pixels of the white portion. It can be estimated that the larger the number of pixels detected as the black-eye portion, the larger the eyes are opened.

  Further, when the number of pixels detected as a black eye portion is very small, it can be estimated that the eyes are closed. For example, 90 (unit: pixel) can be set as the threshold used for the eye open / close determination. This value can be set to an optimal value as appropriate according to the size of the area detected as the one-eye area.

  In the present embodiment, in addition to the eye opening / closing determination, it is also determined whether or not the face is smiling. As described above, it can be estimated that the larger the number of pixels detected as the black-eye portion, the larger the eyes are opened. But just because your face is the most open, you can't say it's a laughing expression. This is because there is a possibility that the corners of the eyes are lowered when smiling, and there is a possibility that the eyes of the smile are in a state where the eyelids are lower than the eyes without expression. Therefore, the eyes from the slightly closed state to the fully opened eye are considered as the smiling eyes. Whether or not the eye is in this state is determined depending on whether or not the number of pixels of the extracted black eye region is within a predetermined range.

For example, it is assumed that n photographed images Po are input, and one eye region of the left eye of a specific person Ps in each photographed image Po is detected. The number of pixels of the black eye region extracted from the detected one eye region is p1, p2,. . . , Pn. At this time, it is determined that the detected eye is a smiling eye when the number p of pixels of the black eye is within the range expressed by the following equation (1).

  That is, the largest number of pixels is detected from among the extracted number of pixels of the black eye region, and ½ or more of the maximum number of pixels is set as a range to be distinguished from smiling eyes.

  FIG. 5 shows a correspondence example between the number of pixels in the black eye region extracted from the one eye region and the estimation result of the eye opening degree. On the left side of FIG. 5, a portion in which the face of the person Ps2 is extracted from the three images of the photographed image Po1 to the photographed image Po3 is extracted from the top to the bottom. Po2 and photographed image Po3 are arranged in this order. The person Ps2 closes his eyes in the photographed image Po1, and opens his eyes in the photographed image Po2 and the photographed image Po3.

  The right side of FIG. 5 shows the black eye region extracted from the one eye region detected from each captured image Po and the number of pixels p. Note that the black eye regions are sorted in descending order from the top to the bottom with the number of pixels as a pair for both eyes.

  The uppermost stage on the right side of FIG. 5 shows the black eye areas extracted from the right eye area Re2r and the left eye area Re2l detected from the captured image Po2 shown in the middle stage on the left side of FIG. It is indicated that the pixel number p of the black eye region extracted from the right eye region Re2r is 243, and the pixel number p of the black eye region extracted from the left eye region Re2l is 235.

  The middle stage shows the black eye areas extracted from the right eye area Re3r and the left eye area Re3l detected from the captured image Po3 shown in the lowermost stage on the left side of FIG. It is indicated that the pixel number p of the black eye region extracted from the right eye region Re3r is 203, and the pixel number p of the black eye region extracted from the left eye region Re3l is 223.

  The bottom row shows the black eye regions extracted from the right eye region Re1r and the left eye region Re1l detected from the photographed image Po1 shown in the top row on the left side of FIG. It is shown that the number of pixels p of the black eye region extracted from the right eye region Re1r is 74, and the number of pixels p of the black eye region extracted from the left eye region Re1l is 25.

  Of the number p of pixels in the black eye region, 243 has the largest value. Therefore, it is possible to estimate an eye having a black eye region having more pixels p than 243/2 = 121.5 as a smiling eye. In the example shown on the right side of FIG. 5, the eyes extracted from the photographed image Po2 shown in the top row and the eyes extracted from the photographed image Po3 shown in the middle row are classified into smiley eyes.

  The number of pixels p of each black eye region extracted from the right eye region Re1r and the left eye region Re1l of the photographed image Po3 is “74” and “25”, respectively, which is smaller than 90 pixels set as the threshold value for eye open / close determination. . Therefore, it is estimated that both the eyes of the right eye region Re1r and the left eye region Re1l of the captured image Po3 are closed. Such a face including closed eyes, that is, the face of the person Ps2 in the photographed image Po3 is not determined to be a smile even if the mouth corner is raised.

  Next, a method for estimating the rising angle of the mouth corner will be described. When people laugh, their mouth corners are generally higher than when they are expressionless. As the corner of the mouth rises, the lip line should be convex downward. In this embodiment, the lip line is approximated to a quadratic function, and the degree of smile is estimated based on the coefficient of the quadratic term. The larger the coefficient of the quadratic term of the quadratic function, the more convex the function curve will be. Therefore, it can be estimated that the mouth region including the lip line having the largest coefficient is the mouth region having the highest mouth angle. That is, it is estimated that the mouth area is laughing.

  In approximating the lip line to a quadratic function, it is necessary to obtain data points having lip line information. In this embodiment, this data point is obtained by detecting the minimum point of each column for the Y (luminance) component of the mouth area. FIG. 6A shows an image in which the lip line is detected, and FIG. 6B is a graph in which Y components detected on the vertical line indicated as the line Ln in FIG. 6A are plotted on the vertical coordinates. It is. The vertical axis in FIG. 6B indicates the value of the Y component (luminance value), and the horizontal axis indicates the coordinates in the vertical (y) direction of the image.

Pixels that hold lip line information have lower brightness values than pixels that hold skin, lip, and tooth information in the same column of the image (for example, line Ln in FIG. 6A). Therefore, when the sequence of images is viewed as a one-dimensional signal, the lip line matches the minimum point. In other words, the lip line pixel can be obtained by detecting the minimum point. In the example shown in FIG. 6B, the luminance value near the y coordinate of “20” is detected as a minimum point. After detecting the pixels of the lip line, these data are approximated to a quadratic function (f (x) = ax ² + bx + c) by the least square method. The larger the value of the coefficient a of the quadratic function, the more convex the function curve is. That is, it can be estimated that the mouth area is a smiling mouth area with a raised corner.

  FIG. 7 is a diagram illustrating the correspondence between the mouth area image in the captured image Po and the detected lip line. On the left side of FIG. 7, a portion in which the face of the same person Ps2 is extracted from the three images of the photographed image Po1 to the photographed image Po3 is extracted in the photographed image Po3 from the top to the bottom. The captured image Po2 and the captured image Po1 are arranged in this order.

The right side of FIG. 7 shows the lip line extracted from the mouth region Rm detected from each captured image Po and the value of the coefficient a of the quadratic function approximating it. The coefficient a of the quadratic function obtained by approximating the lip line detected from the photographed image Po3 is 2.98 × 10 ⁻⁵ , and the coefficient of the quadratic function obtained by approximating the lip line detected from the photographed image Po2 a is 33.3 × 10 ⁻⁵ . The coefficient a of the quadratic function obtained by approximating the lip line detected from the photographed image Po1 is 8.45 × 10 ⁻⁵ . Among the three photographed images Po shown on the left side of FIG. 7, the coefficient a of the quadratic function approximating the lip line extracted from the photographed image Po2 having the highest mouth corner is the largest value. I understand.

  The facial expression is estimated using the information on the degree of opening of the eyes and the information on the degree of rising of the mouth corner. More specifically, first, the photographed image Po is divided into one with eyes open and one with eyes closed. Then, in both closed and open ones, an operation is performed to sort the captured images Po in descending order of the quadratic function coefficient a obtained by approximating the lip line. Then, the sorted captured images Po are arranged in the order of open eyes and closed eyes. Finally, the order determined by this work is given to each face as an evaluation value as it is.

  As a result, a higher evaluation value is given to the captured image Po including a facial expression with wider eyes and a raised mouth corner, and photographing including a facial expression with a closed eye and a lower mouth corner. A lower evaluation value is assigned to the image Po. This evaluation value assignment operation is performed for all persons included in the captured image Po. The evaluation value in each captured image Po can be obtained by summing up the evaluation values of the faces of the persons shown in the image.

  The replacement side / replacement captured image selection unit 122 (see FIG. 3) selects the captured image Po having the highest evaluation value as the replacement captured image that is the basis of the best shot. Then, a captured image including a face area to be replaced with respect to the replacement captured image is selected as a replacement-side captured image. It is assumed that the face of each person Ps in the photographed image Po selected as the replacement photographed image has the highest evaluation value for each person. That is, the best facial expression of each person may be within one photographed image Po. In such a case, subsequent processing is not performed, and the captured image Po is selected as the best shot.

<2-1-2. Details of processing of composite image generation unit>
[2-1-2-1. Details of processing of replacement area range determination unit]
Next, details of processing in each unit constituting the composite image generation unit 130 will be described. First, details of the processing of the replacement region range determination unit 131 will be described with reference to FIG. The replacement region range determination unit 131 calculates the elliptical range of the replacement region extracted by the subsequent multi-resolution decomposition unit 132 based on the distance between the eyes of the face portion detected from the captured image Po.

  The size of the ellipse is preferably a size that includes the entire face region and also includes the entire hair portion. By setting the ellipse to such a size and setting the region surrounded by the ellipse as the replacement region, the hair region is not covered at the boundary between the replacement portion and the replacement portion. This is because, if a hair region is applied to the boundary portion, the boundary portion is recognized as an unnatural image by being decomposed and reconstructed into multiple resolutions in the subsequent processing.

In FIG. 8, the right eye region Rer and the left eye region Rel detected by the eye / mouth part detection unit 112 are indicated by rectangles. A line Ep indicated by an arrow connecting the center of the right eye region Rer and the center of the left eye region Rel corresponds to the pupil interval. The center of the line Ep is set as the center of the ellipse E, and the major axis rL and the minor axis rS of the ellipse E are determined based on this point. The inclination of the ellipse E, that is, the θ angle with respect to the horizontal direction of the major axis rL is positioned to be 0 °. The major axis rL and the minor axis rS of the ellipse E are obtained by the following equations 1 and 2, respectively.
Long diameter rL = 1.5 rhEp Formula 1
Minor axis rS = 1.5 rwEp Equation 2

  “Rh” in the above formula 1 and “rw” in the above formula 2 are the Japanese standard of the ratio of the face height to the pupil distance and the ratio of the face width when the pupil distance is 1, respectively. The typical value is shown. These values were calculated using data described in “Japanese Head Dimensions Database 2001” provided by the National Institute of Advanced Industrial Science and Technology. Specifically, the ratio of the height of the face and the ratio of the width when the pupil interval was set to 1 for men and women of each age were calculated, and the calculated height and width were averaged. As a result, values of face height rh = 3.57 and face width rw = 2.49 were obtained.

  The value of the multiplier “1.5” in the above formulas 1 and 2 is a value calculated with the intention of fitting the hair portion up to the replacement area. Accordingly, different values may be set according to the size of the range to be extracted as the replacement area.

[2-1-2-2. Details of processing of multi-resolution decomposition unit]
Next, details of the processing of the multi-resolution decomposition unit 132 will be described with reference to FIGS. 9 and 10. As described above, in this embodiment, the image in the replacement area is replaced with the image in the replacement area under the discrete wavelet transform area. That is, after the captured image Po is decomposed into multiple resolutions by discrete wavelet transform, the images are replaced under each frequency region obtained by the decomposition. The multi-resolution decomposition unit 132 performs processing for decomposing the captured image Po into multi-resolution by discrete wavelet transform. In this embodiment, the entire replacement-side captured image as the first region and the entire replacement-captured captured image as the second region are decomposed into multiple resolutions by non-decimation wavelet transform, which is a kind of discrete wavelet transform. . The non-decimation wavelet transform is a technique for performing decomposition and reconstruction into multiple resolutions without performing the thinning process (upsampling / downsampling) performed in the discrete wavelet transform.

  The discrete wavelet transform is a kind of subband coding. The discrete wavelet transform of the one-dimensional signal x (n) uses a two-channel filter bank to divide a signal into bands and recursively filter banks for low frequency components. Can be expressed in octave division. FIG. 9 shows a process in which the multi-resolution decomposition unit 132 performs discrete wavelet decomposition of the one-dimensional discrete signal x (n) to three levels.

  “H (z)” in FIG. 9 indicates a low-pass filter, and “G (z)” indicates a high-pass filter. These are used as a set of filters. The low-pass filter H (z) and the high-pass filter G (z) upsample the filter every time one level is decomposed. Since the captured image Po is a two-dimensional signal, this processing is performed in the vertical direction or the horizontal direction, and then the same processing is performed in the other direction. By performing this processing, the captured image Po is decomposed into a low frequency component LL, a vertical high frequency component LH, a horizontal high frequency component HL, and a diagonal high frequency component HH.

  FIG. 10 is a diagram illustrating an example in which an image is decomposed by one level using non-decimated wavelet transform. The result of performing non-decimation wavelet transform on the grayscale LENNA image shown on the left side of FIG. 10 in the horizontal direction and the vertical direction is shown on the right side of FIG. The low frequency component LL, the vertical high frequency component LH, the horizontal high frequency component HL, and the diagonal high frequency component HH are obtained by the non-decimation wavelet transform, as shown in the clockwise direction from the upper left. Since the thinning process is not performed, the size of each frequency component is the same as the size of the original image. When the photographed image Po is a color image, R (red), G (green), and B (blue) color components or luminance signals (Y) and color difference signals (U, V) are respectively non-decimated wavelets. It is necessary to decompose by conversion.

  The non-decimated wavelet transform is repeated a plurality of times and decomposed to n level and then reconstructed, so that the boundary between the replacement portion and the replacement portion in the synthesized image Pc generated as the best shot is less noticeable and natural. Can be. This is because the processing is increased by increasing the decomposition level, so that the boundary line is also smoothed. As a result of experiments, it has been found that a certain effect can be obtained by decomposing to at least three levels.

  When the low-frequency component LL obtained by the one-level decomposition is further decomposed, a low-frequency component LL2, a vertical high-frequency component LH2, a horizontal high-frequency component HL2, and an oblique high-frequency component HH2 are obtained. By further decomposing the low frequency component LL2 obtained here, a low frequency component LL3, a vertical high frequency component LH3, a horizontal high frequency component HL3, and an oblique high frequency component HH3 are obtained. By repeating this process, it is possible to perform decomposition up to a desired n level.

[2-1-2-3. Details of processing in the replacement processing unit]
Next, details of the processing of the replacement processing unit 133 will be described with reference to FIGS. 11 and 12. First, the replacement processing unit 133 uses the ellipse E whose size is determined by the replacement region range determination unit 131 to cut out a replacement region from the replacement-side captured image. Then, the face image in the corresponding area of the replacement photographed image is replaced with the face image in the extracted replacement area.

  FIG. 11 is a diagram illustrating an example of face area synthesis processing in the R (red) LL component Po2LL of the captured image Po2 selected as the replacement captured image. As shown in FIG. 11, the face region Rf31 cut out from the R LL component Po1LL of the replacement photographed image Po1 is arranged at a corresponding position on the LL component Po2LL on the side where the replacement is performed. At this time, alignment is performed so that the center of the face area Rf31 cut out by the ellipse E is arranged at the center position of the pupil interval of the face area on the side where the replacement is performed. The inclination angle θ of the ellipse E is adjusted to be 0 ° with respect to the horizontal direction of the image. After the alignment is completed, a compositing process is performed to replace the face image on the LL component Po2LL on the replacement side with the face image of the face region Rf31 extracted as the replacement region. This replacement process is performed for each frequency component of the R, G, and B color components.

  FIG. 12 shows an example of face area replacement (synthesizing) processing for each frequency component of R, G, and B color components. The left side of FIG. 12 shows each frequency component of the photographed image Po2 selected as the replacement photographed image, and the right side shows each frequency component of the photographed image Po1 selected as the replacement photographed image.

  The replacement processing unit 133 uses the ellipse E whose size is determined by the replacement region range determination unit 131 to calculate a face region Rf31 as a replacement region from each frequency component generated from the captured image Po1 that is the replacement-side captured image. Extract. That is, the face region Rf31 is extracted from each of the low frequency component Po1LL, the vertical high frequency component Po1LH, the horizontal high frequency component Po1HL, and the oblique high frequency component Po1HH.

  Then, a replacement (combination) process of the extracted face region Rf31 is performed on each frequency component generated from the captured image Po2 that is the replacement captured image. That is, for each of the low-frequency component Po2LL, the vertical high-frequency component Po2LH, the horizontal high-frequency component Po2HL, and the oblique high-frequency component Po2HH, the face image of the replacement region corresponding to the replacement region is a face region Rf31 that is the replacement region. Replace with the face image inside. The face image replacement process is performed for the number of persons that need to be replaced. In the example shown in FIG. 1, the process is performed on the person Ps1 and the person Ps3.

[2-1-2-4. Details of processing of the reconstruction unit]
Next, details of the processing in the reconstruction unit 134 will be described with reference to FIG. Also in the reconstruction unit 134 shown in FIG. 13, one set of low-pass filter H (z) and high-pass filter G (z) is assembled in three stages. One set of low-pass filter H (z) and high-pass filter G (z) downsamples the filter every time one level is reconstructed. By passing through the filter assembled in this way, each frequency component decomposed by the multi-resolution decomposition unit 132 is subjected to inverse wavelet transform and reconstructed into one image.

(2-2. Example of image processing method according to first embodiment)
Next, an example of the image processing method according to the present embodiment will be described with reference to the flowchart of FIG. First, a plurality of captured images Po continuously captured by the image capturing apparatus 200 are taken into the image processing apparatus 100 (step S1). Next, the face part detection unit 111 detects a face part in each of the plurality of photographed images Po (step S2), and the eye / mouth part detection unit 112 further detects an eye and mouth part from the face part. It is detected (step S3). Next, the facial expression estimation unit 121 estimates the facial expression of the person Ps based on the eye and mouth part images detected by the eye / mouth part detection unit 112 (step S4).

  Subsequently, based on the facial expression estimated by the facial expression estimation unit 121, the replacement / replaced captured image selection unit 122 shows the best facial expression for each person Ps in the captured image Po. A captured image Po is determined (step S5). Then, it is determined whether or not all the captured images Po including the best facial expression of each person Ps selected in step S5 are the same captured image Po (step S6). If all the captured images Po are the same, the captured image Po is adopted as the best shot (step S7).

  When the photographed images Po including the best facial expressions of each person Ps are not all the same photographed images Po, the replacement photographed image that is the basis of the best shot is obtained by the replacement / replaced photographed image selection unit 122. It is determined (step S8). Specifically, the photographed image Po having the highest total evaluation value of the faces of each person Ps in the photographed image Po is selected as the replacement photographed image.

  Then, the replacement area range determination unit 131 determines the major axis and the minor axis of the ellipse E for cutting out the face area as the replacement area based on the replacement side captured image including the best facial expression of each person Ps in the captured image Po. Calculated (step S9). Subsequently, the multi-resolution decomposition unit 132 performs discrete wavelet transform (non-decimation wavelet transform) on the replacement captured image and the replacement-side captured image (step S10). Then, in each frequency component of the replacement-captured image and each frequency component of the replacement-side captured image, the replacement processing unit 133 extracts a face area using the ellipse E (step S11).

  Subsequently, the replacement processing unit 133 performs a process of replacing the face image in the replacement area in the replacement photographed image with the face image in the replacement area (step S12). Finally, each frequency component of the replacement-captured image on which the face image has been replaced is subjected to inverse wavelet transform by the reconstruction unit 134, thereby generating a composite image as the best shot (step S13).

  According to the present embodiment, replacement of the replacement captured image with the image in the replacement area is performed under the discrete wavelet transform area. For this reason, when an image is reconstructed by inverse wavelet transform, each frequency component is filtered. Thereby, in the reconstructed composite image Pc, the boundary between the replacement portion and the replacement portion becomes inconspicuous.

  Further, according to the present embodiment, an area including a human face portion and a hair portion is cut out by an ellipse E as a replacement region. Thereby, for example, the facial expression of the person Ps in the composite image Pc becomes natural as compared with a case where only the eye region or only the mouth region is replaced. Further, since the replacement area is cut out by the ellipse E instead of the rectangle, the boundary between the replacement portion and the replacement portion in the composite image Pc becomes more inconspicuous than in the case of cutting out the rectangle.

  FIG. 15 shows a composite image synthesized in the spatial domain without performing the decomposition by the discrete wavelet transform, and a composite image Pc according to the present embodiment. FIG. 15A is a composite image generated by a conventional method, and FIG. 15B is a composite image Pc generated by the present embodiment. A region from the cheek of the person Ps to the chin is shown in the upper left corner of the images of FIGS. 15A and 15B, and a shoulder portion of the person Ps is shown below the images.

  In the synthesized image by the conventional method shown in FIG. 15A, a linear boundary line from the upper right to the lower left can be visually recognized between the area around the face and the background area outside the face. On the other hand, such a boundary line is not seen in the composite image Pc according to the present embodiment shown in FIG. 15B. That is, it can be seen that a more natural composite image Pc is obtained.

[3. Second Embodiment]
(3-1. Overview of Second Embodiment)
In the above-described first embodiment, when the images are combined under the discrete wavelet transform region, the entire replacement-captured captured image and replacement-side captured image are decomposed into multiple resolutions. In the present embodiment example, only a partial area of each of the replacement captured image and the replacement side captured image is decomposed into multiple resolutions and reconstructed. As a result, the amount of data required for processing is reduced, and the processing time can also be shortened.

  FIG. 16 is a diagram showing an area Ar where decomposition and reconstruction are performed into multiple resolutions according to the present embodiment. The region Ar is a region formed around the outer periphery of the replacement region, and is defined as a donut-shaped region having a predetermined width on the inner peripheral side and the outer peripheral side around the ellipse E indicating the range of the replacement region. .

  In the present embodiment example, this area Ar is extracted from both the replacement photographed image and the replacement photographed image. FIG. 17A is a diagram illustrating a region Ar31 (first region) extracted from the captured image Po1 as the replacement-side captured image, and FIG. 7B is a region extracted from the captured image Po2 as the replacement captured image. It is the figure which showed Ar32 (2nd area | region). In any of the drawings, both the region Ar31 and the region Ar32 are indicated by a one-dot chain line.

  The multi-resolution decomposition unit 132 performs non-decimation wavelet transform on the image in the area Ar31 and the image in the area Ar32 set in this way, and decomposes them into frequency components. The replacement processing unit 133 performs a process of replacing a part of the image in the replacement area Ar32 with a part of the image in the replacement area Ar31.

  FIG. 18 is a diagram illustrating an example of replacement processing by the replacement processing unit 133. In the upper left of FIG. 18A, the low-frequency component Ar32LL of the area Ar32 around the outer periphery of the face area that is not a good expression shown in FIG. In the lower right of FIG. 18A, only the inner peripheral portion (inside the ellipse E) of the low-frequency component Ar31LL of the region Ar31 around the outer periphery of the face region of good expression shown in FIG. . The replacement processing unit 133 according to the present embodiment replaces the inner peripheral portion of the low-frequency component Ar32LL generated from the replacement captured image with the inner peripheral portion of the low-frequency component Ar31LL generated from the replacement-side captured image. . Thereby, as shown in FIG. 18B, the low-frequency component Ar32LL generated from the replacement-captured image and the low-frequency component Ar31LL generated from the replacement-side captured image are combined.

  The synthesized low-frequency component Ar31LL and low-frequency component Ar32LL are subjected to inverse wavelet transform by the reconstruction unit 134 and reconstructed into the region Ar ′.

  In this embodiment, as shown in FIG. 19, the image of the face area Rf31 in the ellipse E, the image of the area Ar ′ reconstructed after replacement, and the background image of the captured image Po2 that is the replacement captured image Is combined to obtain a composite image Pc as the best shot.

(3-2. Configuration Example of Image Processing Device)
FIG. 20 is a diagram illustrating a configuration example of an image processing apparatus 100A according to the present embodiment. 20, portions corresponding to those in FIG. 3 are denoted by the same reference numerals, and redundant description is omitted. The difference in configuration from the image processing apparatus 100 according to the first embodiment shown in FIG. 3 is the internal configuration of the composite image generation unit 130A. In the present embodiment, a composite image generation processing unit 135 is further added after the reconstruction unit 134. The composite image generation processing unit 135 includes an image in the area Ar ′ that has been decomposed into multiple resolutions by the multi-resolution decomposition unit 132 and reconstructed by the reconstruction unit 134, a face image in the replacement area, and a replacement photographed image. A synthesized image Pc is generated by synthesizing the background image.

  Note that the width of the area Ar extracted from both the replacement photographic image and the replacement photographic image shown in FIG. 16 needs to be at least 1.5 L on the inner peripheral side and the outer peripheral side with the ellipse E as the center. is there. Here, “L” is the filter length of the non-decimated wavelet transform performed by the multiresolution decomposition unit 132. The width of the region Ar may be increased as much as it is wider than this width.

Here, the reason why at least 3L is required as the width of the region Ar will be described. For example, when the coefficient of a general filter having a filter length L of 5 is “h (n)” and the input to the filter is “x (n)”, the output y (n) from the filter is expressed by the following equation: 3 can be indicated.
y (n) = h (−2) × x (n + 2) + h (−1) × x (n + 1) + h (0) × x (n) + h (1) × x (n−1) + h (2) × x (n-2) Equation 3

  That is, the output y (n) at the current time n is affected by the input in the range of n ± L / 2 (2.5 here). In other words, the input x (n) at the current time n affects the output y (n) in the range of n ± L / 2.

FIG. 21A is a block diagram showing processing from non-decimated wavelet transform to reconstruction according to the present invention. The length L of the non-decimated wavelet transform filter is represented by (1 + 2 + 4 +... +2 ⁿ⁻¹ ) N + 1. Here, “N” is the order of the low-pass filter H (z) and the high-pass filter G (z). As shown in FIG. 21A, the input is “x (n)”, the output after non-decimated wavelet transform (uDWT) is “u (n)”, the output after replacement synthesis processing is “v (n)”, and the inverse wavelet The converted output is assumed to be “y (n)”.

  FIG. 21B shows the magnitude of the luminance value at each pixel position indicated by “k−2” to “k + 2” of the frequency component u1 (n) obtained by decomposing the replacement-side captured image, and the length of the arrow. It is the figure shown by. FIG. 21C shows the magnitude of the luminance value at each pixel position indicated by “k−2” to “k + 2” of the frequency component u2 (n) obtained by decomposing the replacement photographed image, and the length of the arrow. It is the figure shown by. FIG. 21D is a diagram illustrating the magnitude of the luminance value at each pixel position indicated by “k−2” to “k + 2” in the output v (n) after the replacement synthesis process, by the length of the arrow. .

  For example, it is assumed that image replacement and composition are performed at a position between “k−1” and “k” indicated by a vertical line in FIG. 21D. In this case, the output y (n) in the range of k ± L / 2 is affected by the replacement and synthesis processing. In order to obtain the output y (n) in the range of k ± L / 2, it is necessary to input the luminance value in the range of k ± L / 2 ± L / 2, that is, k ± L, as the input v (n). is there. Further, in order to calculate u (n) in the range of k ± L, it is necessary to input a luminance value in the range of k ± L ± L / 2, that is, k ± 3L / 2, as the input x (n). is there.

  Therefore, in order to perform the non-decimated wavelet transform process, the image replacement and synthesis process, and the inverse wavelet transform process, the ellipse E arranged at the position of k is centered at ± 3 L / 2, that is, ± 1.5 L. A range of pixel values is required as input. For this reason, when replacing the image in the replacement area corresponding to the replacement area with the image in the replacement area cut out by the ellipse E, at least 1.5 L each in the inner circumferential direction and the outer circumferential direction with the ellipse E as the center. It is necessary to input a region Ar having a width to the multi-resolution decomposition unit 132.

(3-3. Example of image processing method)
Next, an image processing method according to the present embodiment will be described with reference to the flowchart of FIG. The image processing method according to this embodiment is common to the image processing according to the first embodiment shown in FIG. 14 from step S1 to step S6. Therefore, FIG. 22 shows processing after step S6 of FIG.

  First, the replacement side / replacement captured image selection unit 122 determines a replacement captured image that is the basis of the best shot (step S21). Then, the replacement area range determination unit 131 calculates the major axis and the minor axis of the ellipse E for cutting out the face area as the replacement area (step S22). Further, the replacement area range determination unit 131 determines areas Ar having a width of 1.5 L on the inner circumference side and the outer circumference side of the ellipse E around the ellipse E (step S23).

  Next, the multi-resolution decomposition unit 132 extracts a region Ar from both the replacement-captured captured image and the replacement-side captured image, and the extracted region Ar is subjected to discrete wavelet transform (non-decimated wavelet transform) (step S24). Subsequently, in each frequency component decomposed into multiple resolutions by the replacement processing unit 133, an image in the inner peripheral area of the ellipse E in the replacement-side captured image, and the inner periphery of the corresponding area in the replacement-captured image A process of replacing the image in the side area is performed (step S25).

  Then, the reconstruction unit 134 performs inverse wavelet transform on the image of the area Ar ′ in each frequency component for which replacement has been performed, and reconstructs the image (step S26). Lastly, the composite image generation processing unit 135 uses the face image in the replacement area cut out by the ellipse E, the reconstructed area Ar ′ around the face image, and the background image of the replacement photographed image. Thus, the composite image Pc as the best shot is generated (step S27).

  According to the second embodiment described above, by replacing the ± 1.5L region Ar centered on the ellipse E under the discrete wavelet transform region, the boundary between the replacement portion and the replacement portion is inconspicuous. A natural composite image can be obtained.

  In addition, since the area subjected to discrete wavelet transform is limited to an area of ± 1.5 L centered on the ellipse E, the calculation time is shortened and the time until the composite image Pc as the best shot is generated is also shortened. Become. Therefore, for example, even when the captured image Po is very high definition and the amount of data is large, the processing can be performed at high speed.

[4. Various modifications)
In each of the above-described embodiments, the example in which the multi-resolution decomposition unit 132 performs the non-decimation wavelet transform is described. However, the present invention is not limited to this, and normal discrete wavelet transform may be performed.

  Further, in each of the above-described embodiments, an example in which an area including the entire face area is extracted as a replacement area has been described, but the present invention is not limited to this. Only the eye part and the mouth part may be extracted as replacement regions.

  In each of the above-described embodiments, a face image including a good expression is extracted as a replacement area and a face image that does not have a good expression is replaced. However, the present invention is not limited to this. Regardless of whether the expression is good or bad, an image in an arbitrary area may be replaced with an image in a corresponding area.

[5. Configuration example of imaging apparatus according to an embodiment of the present invention]
FIG. 23 is a block diagram illustrating a configuration example of the imaging apparatus 200A when the image processing apparatus of the present invention is applied to the imaging apparatus. An imaging apparatus 200A illustrated in FIG. 23 has a continuous shooting function and generates a composite image Pc as a best shot using a plurality of captured images Po obtained by continuous shooting. The imaging apparatus 200A includes a lens 201, an imaging element 202, a signal processing unit 203, an image processing unit 204, and a timing generator (TG) 205.

  The lens 201 includes a single lens or a plurality of lens groups, and takes in subject light into the imaging device 200. An imaging element 202 as an imaging unit photoelectrically converts subject light incident through the lens 201 to generate an image signal. The image sensor 202 is configured by, for example, a CCD (Charge Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, or the like.

  The signal processing unit 203 performs various signal processes such as AGC (Auto Gain Control), AWB (Auto White Balance), and gamma correction on the image signal obtained by the image sensor 202. The signal processing unit 203 also performs processing for converting the format of the image signal input in R, G, and B according to the output format. When the image signal is output in the YUV format, the multi-resolution decomposition process performed in the next image processing unit 204 is the same as in the case of the RGB signal, the luminance signal, the color difference signal (Cb), and the color difference signal (Cr ) For each of these.

  The image processing unit 204 performs color process processing such as demosaicing and color space conversion, resolution conversion processing, image filter processing, and the like on the image signal subjected to signal processing by the signal processing unit 203. Furthermore, a process of obtaining a composite image Pc as a best shot by combining a plurality of captured images Po is also performed. A timing generator (TG) 205 generates drive timing pulses for the image sensor 202 and pulses for the signal processing unit 203 and the image processing unit 204 and supplies them to these units.

  The imaging apparatus 200A also includes a memory 206 as a storage unit and a recording / playback processing unit (CODEC) 207. The memory 206 includes a ROM (Read Only Memory) and a RAM (Random Access Memory). The RAM is replaced with data in the middle of work, for example, each frequency component generated by the multi-resolution decomposition unit 132 or a replacement processing unit 133. An image to be processed is temporarily stored. The ROM stores a captured image Po, a moving image, and the like captured by the image sensor 202. The recording / playback processing unit 207 performs a process of compressing the image signal subjected to the image processing by the image processing unit 204 and writing the compressed image data in the memory 206, or reading and decompressing the compressed data stored in the memory 206.

  The imaging apparatus 200 </ b> A includes a digital / analog conversion unit (D / A) 208, a video encoder 209, and a display unit 210. The D / A 208 converts the digital image signal subjected to image processing by the image processing unit 204 into an analog image signal. The video encoder 209 encodes the image signal converted into an analog signal by the D / A 208 into a moving image and outputs the encoded image signal to the display unit 210. The display unit 210 is a display that displays a video encoded by the video encoder 209, and includes a LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

  Furthermore, the imaging apparatus 200 </ b> A includes a control unit 211, an external storage medium 212, an external storage medium I / F unit 213, and an operation input unit 214. The control unit 211 includes a CPU (Central Processing Unit) and the like, and controls each unit configuring the imaging apparatus 200A. The external storage medium 212 refers to a removable storage medium called a removable medium, and an external storage medium such as a DVD (Digital Versatile Disc) or HDD (Hard Disc Drive). The external storage medium I / F unit 213 controls reading and writing of data with respect to the external storage medium 212. The operation input unit 214 includes buttons, knobs, keys, and the like. The operation input unit 214 converts the operation content input by the user into an operation signal and outputs the operation signal to the control unit 211.

  FIG. 23 shows an example in which the image processing apparatus of the present invention is applied to an imaging apparatus 200A including a display unit 210. However, an image signal is not transmitted to an external display device without including the display unit 210. It may be applied to an imaging device that outputs.

  The image processing apparatus of the present invention is not limited to an image processing apparatus used in a stand-alone environment. For example, the present invention may be applied to an image processing apparatus used in an image processing service that processes and develops a photographed image Po transmitted via a network, or processes it into a photo book or the like.

  The series of processing in the above-described embodiment can be executed by software. When the series of processing is executed by software, the processing can be executed by a computer in which a program constituting the software is incorporated in dedicated hardware or a computer in which programs for executing various functions are installed. For example, a program constituting desired software may be installed and executed on a general-purpose personal computer or the like. Further, the present invention may be applied to software placed on a server connected to a network or in a cloud computing environment.

  Further, a recording medium storing software program codes for realizing the functions of the above-described embodiments may be supplied to the system or apparatus. It goes without saying that the function is also realized by reading and executing the program code stored in the recording medium by a computer (or a control device such as a CPU) of the system or apparatus.

  DESCRIPTION OF SYMBOLS 100,100A ... Image processing apparatus, 110 ... Face detection part, 111 ... Face part detection part, 112 ... Eye / mouth part detection part, 120 ... Photographed image selection part, 121 ... Expression estimation part, 122 ... Replacement side / substitution Captured image selection unit, 130, 130A ... composite image generation unit, 131 ... replacement area range determination unit, 132 ... multi-resolution decomposition unit, 133 ... replacement processing unit, 134 ... reconstruction unit, 135 ... composite image generation processing unit, 200 , 200A ... imaging device, 201 ... lens, 202 ... imaging device, 203 ... signal processing unit, 204 ... image processing unit, 205 ... timing generator, 206 ... memory, 207 ... digital / analog conversion unit, 208 ... recording / reproduction processing unit 209: Video encoder 210 ... Display unit 211 ... Control unit 212 ... External storage medium 213 ... External storage medium I / F unit 214 ... Operation input Part

Claims (10)

A photographed image selection unit that selects a replacement photographed image on which the image is replaced from a plurality of photographed images that are continuously photographed, and a replacement-side photographed image on the replacement side of the image;
A first region including at least a replacement region outer peripheral region that is a region around the outer periphery of a replacement region in the replacement-side captured image, and a replacement region that is a region around the outer periphery of the replacement region in the replacement captured image A multi-resolution decomposition unit that decomposes the second region including at least the outer peripheral region into multi-resolution;
A replacement processing unit that replaces at least a part of the second region with at least a part of the first region in each frequency component obtained by the decomposition by the multi-resolution decomposition unit;
A first process for reconstructing each frequency component replaced by the replacement processing unit, or a second process for combining the reconstructed image obtained by the first process and the image in the replacement area. processed by, and a composite image generating unit that generates a composite image,
When the first area is the outer peripheral area of the replacement area, the multi-resolution decomposition unit performs the outer peripheral area of the replacement area as the first area and the outer periphery of the replacement area as the second area. Each of the peripheral regions is decomposed into multiple resolutions, and the replacement processing unit is configured to decompose each frequency component in the peripheral region around the replacement region and each frequency in the peripheral region around the replacement region obtained by the multi-resolution decomposition unit. In the component, the inner periphery side region of the replacement region outer periphery peripheral region is replaced with the inner periphery side region of the replacement region outer periphery peripheral region, the composite image generation unit generates the composite image by the second processing,
When the first region is the entire replacement-side captured image, the multi-resolution decomposition unit performs the entire replacement-side captured image as the first region and the replacement captured image as the second region. The whole is divided into multiple resolutions, the replacement processing unit, in each frequency component of the entire replacement-side captured image obtained by the decomposition in the multi-resolution decomposition unit and each frequency component of the entire replacement captured image, The image processing device that replaces the replacement area with the replacement area, and wherein the composite image generation unit generates the composite image by the first processing. The replacement region outer periphery peripheral region and the replacement region outer periphery peripheral region are regions formed on the inner periphery side and the outer periphery side around the outer periphery of the replacement region and the outer periphery of the replacement region,
Wherein the A and peripheral side the inner peripheral side region formed in the outer circumferential region which is formed on the outer peripheral side, respectively, the filter at least 1.5 times the length of the length of the filter used in the multiresolution decomposition section The image processing apparatus according to claim 1. The image processing apparatus according to claim 2, wherein the multi-resolution decomposition unit decomposes the first area and the second area into multi-resolution by discrete wavelet transform. A face detection unit for detecting a face part of a person and a part of the face part from the plurality of photographed images;
A facial expression estimation unit that analyzes an image of the facial part detected by the face detection unit and estimates the facial expression;
The photographed image selection unit selects, as the replacement-side photographed image, the photographed image including at least the part of the face whose facial expression is estimated to be good by the facial expression estimation unit, and is not selected as the replacement-side photographed image The image processing apparatus according to claim 3, wherein the replacement photographed image is selected from the photographed images. A replacement region range determination unit for determining a range to be extracted as the replacement region from the replacement-side captured image;
The replacement area range determination unit determines a predetermined range that includes the entire face portion detected by the face detection unit and is wider than the entire face portion as a range to be extracted as the replacement region. Image processing apparatus. The replacement area range determination unit has a minor axis obtained by multiplying the width and length of the face calculated by the face detection unit based on the distance between the parts of both eyes of the face by at least 1.5 times, respectively. The image processing apparatus according to claim 5, wherein a range surrounded by an ellipse having a major axis is determined as a range to be extracted as the replacement region. The image processing apparatus according to claim 5, wherein the multi-resolution decomposition unit decomposes the first area and the second area into multi-resolution by non-decimation wavelet transform . A photographed image selection step of selecting a replacement photographed image on the side where image replacement is performed, and a replacement-side photographed image on the side replacing the image, from among a plurality of photographed images continuously photographed;
A first region including at least a replacement region outer peripheral region that is a region around the outer periphery of a replacement region in the replacement-side captured image, and a replacement region that is a region around the outer periphery of the replacement region in the replacement captured image A multi-resolution decomposition step of decomposing the second region including at least the outer peripheral region into multi-resolution;
A replacement processing step of replacing at least a part of the second region with at least a part of the first region in each frequency component obtained by decomposing into the multi-resolution;
By the first process for reconstructing each frequency component subjected to the replacement, or the second process for synthesizing the reconstructed image obtained by the first process and the image in the replacement region, A composite image generation step of generating a composite image,
When the first area is the perimeter area of the replacement area, the multiresolution decomposition step includes the perimeter area of the replacement area as the first area and the perimeter of the replacement area as the second area. Each of the peripheral regions is decomposed into multiple resolutions, and in the replacement processing step, each frequency component in the peripheral region around the replacement region and each frequency component in the peripheral region around the replacement region obtained by the decomposition into the multi-resolution are obtained. The inner peripheral side region of the replacement region outer peripheral peripheral region is replaced with the inner peripheral side region of the replacement region outer peripheral peripheral region, and in the composite image generation step, the composite image is generated by the second process,
When the first area is the entire replacement-side captured image, in the multi-resolution decomposition step, the entire replacement-side captured image as the first area and the replacement captured image as the second area In the replacement processing step, the frequency component of the entire replacement-side captured image and the frequency component of the entire replacement-captured image obtained by decomposing into the multi-resolution are converted in the replacement processing step. An image processing method in which a replacement area is replaced with the replacement area, and in the composite image generation step, the composite image is generated by the first process. A photographed image selection step of selecting a replacement photographed image on the side where image replacement is performed, and a replacement-side photographed image on the side replacing the image, from among a plurality of photographed images continuously photographed;
A first region including at least a replacement region outer peripheral region that is a region around the outer periphery of a replacement region in the replacement-side captured image, and a replacement region that is a region around the outer periphery of the replacement region in the replacement captured image A multi-resolution decomposition step of decomposing the second region including at least the outer peripheral region into multi-resolution;
A replacement processing step of replacing at least a part of the second region with at least a part of the first region in each frequency component obtained by decomposing into the multi-resolution;
By the first process for reconstructing each frequency component subjected to the replacement, or the second process for synthesizing the reconstructed image obtained by the first process and the image in the replacement region, Causing a computer to execute a composite image generation step of generating a composite image;
When the first area is the perimeter area of the replacement area, the multiresolution decomposition step includes the perimeter area of the replacement area as the first area and the perimeter of the replacement area as the second area. Each of the peripheral regions is decomposed into multiple resolutions, and in the replacement processing step, each frequency component in the peripheral region around the replacement region and each frequency component in the peripheral region around the replacement region obtained by the decomposition into the multi-resolution are obtained. The inner peripheral side region of the replacement region outer peripheral peripheral region is replaced with the inner peripheral side region of the replacement region outer peripheral peripheral region, and in the composite image generation step, the composite image is generated by the second process,
When the first area is the entire replacement-side captured image, in the multi-resolution decomposition step, the entire replacement-side captured image as the first area and the replacement captured image as the second area In the replacement processing step, the frequency component of the entire replacement-side captured image and the frequency component of the entire replacement-captured image obtained by decomposing into the multi-resolution are converted in the replacement processing step. A replacement area is replaced with the replacement area, and in the composite image generation step, the composite image is generated by the first processing.
program. An imaging unit that continuously captures a plurality of captured images;
A photographed image selection unit that selects a replacement photographed image on the side on which image replacement is performed and a replacement-side photographed image on the side that replaces the image from among a plurality of photographed images photographed by the imaging unit;
A first region including at least a replacement region outer peripheral region that is a region around the outer periphery of a replacement region in the replacement-side captured image, and a replacement region that is a region around the outer periphery of the replacement region in the replacement captured image A multi-resolution decomposition unit that decomposes the second region including at least the outer peripheral region into multi-resolution;
A replacement processing unit that replaces at least a part of the second region with at least a part of the first region in each frequency component obtained by the decomposition by the multi-resolution decomposition unit;
A first process for reconstructing each frequency component replaced by the replacement processing unit, or a second process for combining the reconstructed image obtained by the first process and the image in the replacement area. A composite image generation unit that generates a composite image by the processing of
A storage unit for storing the composite image generated by the composite image generation unit,
When the first area is the outer peripheral area of the replacement area, the multi-resolution decomposition unit performs the outer peripheral area of the replacement area as the first area and the outer periphery of the replacement area as the second area. Each of the peripheral regions is decomposed into multiple resolutions, and the replacement processing unit is configured to decompose each frequency component in the peripheral region around the replacement region and each frequency in the peripheral region around the replacement region obtained by the multi-resolution decomposition unit. In the component, the inner periphery side region of the replacement region outer periphery peripheral region is replaced with the inner periphery side region of the replacement region outer periphery peripheral region, the composite image generation unit generates the composite image by the second processing,
When the first region is the entire replacement-side captured image, the multi-resolution decomposition unit performs the entire replacement-side captured image as the first region and the replacement captured image as the second region. The whole is divided into multiple resolutions, the replacement processing unit, in each frequency component of the entire replacement-side captured image obtained by the decomposition in the multi-resolution decomposition unit and each frequency component of the entire replacement captured image, The imaging apparatus , wherein the replacement area is replaced with the replacement area, and the composite image generation unit generates the composite image by the first processing .