JP6676563B2 - Image processing apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to a technique for extracting an arbitrary area in an image.

  In a wide range of fields, there is a demand to extract an arbitrary region from an image. In a television station, a studio, or the like, a chroma key technology for extracting a subject region only by placing a subject on a green background is utilized. By extracting only the subject area, it is possible to obtain added value, for example, by pasting the extracted subject image to another image to generate an image that cannot be realized in reality.

  It is difficult to install a special screen for chroma key photographing in a competition venue or an event venue when it is assumed that a subject is a player in competition, a player performing live performance, an actor on the stage, or the like. As a technique for extracting a subject from an arbitrary background image, there are techniques called segmentation and alpha matting (see Non-Patent Documents 1 and 2). These methods provide a foreground region that is clear to be the foreground, a background region that is clear to be the background, and an unknown region that precisely defines whether the image is the foreground or the background to the image to be processed. The foreground of the unknown area is extracted.

Hiromu Miyashita, Hirohiro Takeuchi, Mariko Yamaguchi, Hidenobu Nagata, Akira Ono, "High-speed and high-precision object extraction using sensors and cameras", IEICE Technical Report, IEICE, 2016 May 26, Vol. 116, No. 73, pp. 17-22 J. Wang and M. Cohen, "Optimized color sampling for robust matting," in Proc. Of IEEE CVPR, IEEE, 2007 Eduardo S. L. Gastal and Manuel M. Oliveira, "Shared Sampling for Real-Time Alpha Matting," Eurographics, 2010, Volume 29, Number 2

  In the segmentation (hereinafter, referred to as NNCL) using the Nearest Neighbor method of Non-Patent Document 1, pixels in an unknown area are classified into foreground or background in terms of color and distance by referring to surrounding foreground and background pixels. The segmentation has a problem that it is difficult to obtain a fine boundary that reproduces the texture of the real object depending on the contrast and texture of the boundary. When an attempt is made to extract a subject from a boundary having a weak contrast with the background or an extremely complicated boundary such as a hair, accuracy may decrease. For such a boundary, post processing is performed to give a transparency called an α value to the boundary portion by an alpha matting method or the like, and to devise the boundary so that it looks natural.

  In alpha matting, an object is extracted so as to look as natural as possible to human eyes by giving a transparency called an α value to pixels in an unknown area. In Robust Matching of the cited document 2, the α value is obtained by using both the color sampling method and the belief propagation method. The color sampling method is a method of obtaining an α value by collecting pixels that are already known as foreground and background as samples. In Robust Matching, samples are taken at intervals to create a more effective sample set for these algorithms. The belief propagation method is a method of obtaining the α value by solving the energy minimization problem of the graph on the assumption that the α value is continuous. In Robust Matching, a graph problem is solved using a random walk method. Robust Matching can obtain a relatively accurate α value, but has a problem that it takes time to process.

  Non-Patent Document 3 shares a candidate point corresponding to a pixel for which an α value is to be calculated in order to speed up a method of giving an α value to a pixel. Specifically, in formulating candidate points for the target pixel in the color sampling method, pixels in the unknown region are classified into a very small set based on the difference between adjacent pixels, and the radiation of the target pixel in the set is determined. Pixels on a straight line having a constant angle are set as candidate points, and the set of candidate points is shared as a set, thereby reducing the number of processes and increasing the speed.

  However, in this method, since the candidate points are scanned in a radial pattern, pixels on the contour physically separated from the target pixel and not selected by the original color sampling method are also selected as candidate points, and the accuracy of the processing result is reduced. There is a problem that falls. In addition, it is necessary to prevent giving an inappropriate transmittance to the pixel caused by sharing the candidate point. In Non-Patent Document 3, based on the difference between the target pixel and an adjacent pixel, A process for evaluating whether the candidate points can be shared is adopted. This process is complicated and has a problem that it is difficult to improve the processing speed.

  The present invention has been made in view of the above, and has as its object to extract an arbitrary region from an image more quickly and precisely.

An image processing apparatus according to a first aspect of the present invention includes: an input unit configured to input a processing target image and information obtained by classifying the processing target image into a foreground area, a background area, and an unknown area; Area determining means for classifying the unknown area into a first area and a second area based on the first area; performing segmentation on the first area; and classifying each pixel of the first area into the foreground area and the background area A segmentation processing unit, a matting processing unit that performs alpha matting on the second region, and obtains a transmittance of each pixel in the second region, and a process on the foreground region, the background region, and the first region and having and an image extracting means for extracting a foreground image from the processed image using the result as a mask image obtained by integrating the processing result for the second region

An image processing method according to a second aspect of the present invention is an image processing method executed by an image processing apparatus, wherein a step of inputting a processing target image and information obtained by classifying the processing target image into a foreground area, a background area, and an unknown area. Classifying the unknown region into a first region and a second region based on a feature amount obtained from the processing target image; performing segmentation on the first region; Classifying the foreground area and the background area; performing alpha matting on the second area to determine the transmittance of each pixel in the second area; characterized in that it comprises the steps of extracting the foreground image from the processed image using the mask image which integrates the processing results for the second region and the processing result for the first area, the To.

  An image processing program according to a third aspect of the present invention is characterized by causing a computer to operate as each unit of the image processing apparatus.

  ADVANTAGE OF THE INVENTION According to this invention, an arbitrary area | region can be extracted from an image more quickly and precisely.

FIG. 2 is a functional block diagram illustrating a configuration of the image processing apparatus according to the embodiment. FIG. 3 is a diagram illustrating a part of an original image input to the image processing apparatus according to the embodiment. FIG. 3 is a diagram illustrating an example of TRIMAP corresponding to FIG. 2. FIG. 4 is a diagram illustrating an example of QUADMAP in which the unknown area in FIG. 3 is classified. 5 is a flowchart illustrating a flow of processing of the image processing apparatus according to the embodiment. It is a flowchart which shows the flow of an area determination process. FIG. 4 is a diagram illustrating an example of a differential image obtained by differentiating a luminance image of an original image. It is a figure showing the example of the threshold image which extracted the pixel whose pixel value is more than the threshold from the differential image. FIG. 14 is a diagram illustrating an example in which a part of a threshold image is divided into small regions. It is a figure showing an example which classified an unknown area for every small area. It is a figure explaining another area discrimination processing. It is a flowchart which shows the flow of a matting process. It is a flowchart which shows the flow of a process which extracts a contour line pixel. FIG. 3 is a diagram illustrating pixels at a boundary between a background region and an unknown region. It is a figure showing the pixel extracted as a contour line pixel of a background field. It is a flowchart which shows the flow of a process which calculates | requires a candidate point. It is a figure explaining selection of a candidate point.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[Configuration of Image Processing Apparatus]
FIG. 1 is a functional block diagram illustrating a configuration of the image processing apparatus according to the present embodiment. The image processing apparatus 1 illustrated in FIG. 1 includes an input unit 11, an area determination unit 12, a segmentation processing unit 13, a matting processing unit 14, an image integration unit 15, and an output unit 16. Each unit included in the image processing apparatus 1 may be configured by a computer including an arithmetic processing unit, a storage device, and the like, and the processing of each unit may be executed by a program. This program is stored in a storage device included in the image processing apparatus 1, and can be recorded on a recording medium such as a magnetic disk, an optical disk, or a semiconductor memory, or can be provided through a network.

  The input unit 11 inputs an original image to be processed and a label image (TRIMAP) in which foreground, background, or an unknown label is assigned to each pixel of the original image. TRIMAP may be created manually by a person, or automatically created from a camera image and a sensing result of a sensor such as a depth sensor or a temperature sensor.

  FIG. 2 shows a part of the original image, and FIG. 3 shows an example of TRIMAP corresponding to the original image in FIG. In the TRIMAP of FIG. 3, pixels to which a foreground label is assigned are indicated by white, pixels to which a background label is assigned are indicated by black, and pixels to which an unknown label is assigned are indicated by gray. Hereinafter, a set of pixels with a foreground label is referred to as a foreground area F, a set of pixels with a background label is referred to as a background area B, and a set of pixels with an unknown label is referred to as an unknown area U.

Region discriminator 12, based on the feature quantity that can be acquired from the original image, an unknown region U trimap, for each partial region, a segmentation processing region U _S and matting process for performing segmentation process performed on the matting treatment region U _M classified, foreground region F, the background region B, and outputs the segmentation process area _{U S,} and QUADMAP comprised of matting processing region _{U M.}

FIG. 4 shows an example of QUADMAP. In QUADMAP in FIG 4, the unknown region U TRIMAP of FIG. 3 is classified into the segmentation processing region _{U S} and matting processing region _{U M.} In the present embodiment, the region determination unit 12, 255 the pixel values of the foreground region F, 0 pixel values of the background region B, and the pixel values of the regions classified as segmentation processing region U _S 98, matting processing region U _M QUADMAP with the pixel value of the area classified as "170" as 170 is output.

  As the feature amount used by the area determination unit 12 to classify the unknown area U, for example, a gradient (edge) of luminance of an image, a frequency component, and the like can be considered. Any feature amount may be used as long as the unknown region U can be divided into a simple region and a complicated region. The simple region is a region in which the difference between the images obtained by the Robust Matching process and the segmentation process is inconspicuous. The complex area is an area in which the transparency of the subject changes gradually at the boundary between the foreground and the background after the Robust Matching processing, and a fine boundary cannot be obtained by the segmentation processing.

Segmentation processing section 13 inputs the original image and QUADMAP, performs segmentation against segmentation processing region U _S, imparts a label foreground or background with respect to pixels included in the segmentation process area U _S. Although any segmentation technique may be used in the segmentation processing unit 13, it is preferable to use NNCL in terms of precision and high speed.

Matting processor 14 inputs the original image and QUADMAP, performs matting process using the Robust Matting method against matting processing region _{U M,} each of the transmission of the pixels included in the matting treatment region _{U M} Determine the degree (α value).

  The image integration unit 15 integrates the processing results of the segmentation processing unit 13 and the matting processing unit 14 to generate a mask image, performs mask processing on the original image using the generated mask image, A foreground image is generated by extracting the foreground from.

  The output unit 16 outputs a foreground image.

[Processing Flow of Image Processing Apparatus]
Next, a flow of processing of the image processing apparatus according to the present embodiment will be described.

  FIG. 5 is a flowchart illustrating a flow of processing of the image processing apparatus according to the present embodiment.

  The input unit 11 inputs an original image and TRIMAP (step S1).

Region discriminator 12 classifies the unknown region U TRIMAP segmentation processing region _{U S} and matting processing region _{U M,} and outputs the QUADMAP (step S2). Details of the processing of the area determination unit 12 will be described later.

Segmentation processing section 13 inputs the original image and QUADMAP, performs segmentation against segmentation processing region _{U S} of QUADMAP (step S3). In the present embodiment, a NNCL, foreground for each pixel in the segmentation process area U _S (pixel value 255) or a background (pixel value 0) to impart label.

Matting processor 14 inputs the original image and QUADMAP, performs matting process using the Robust Matting method against matting processing region _{U M} (step S4). Details of the processing of the matting processing unit 14 will be described later. Matting processing unit 14 later so as not to have trouble when the masking, at the boundary of the segmentation process region U _S and matting processing region U _M, extend two pixels matting treatment region U _M.

  As shown in FIG. 5, the segmentation process and the matting process may be performed in parallel, or the matting process may be performed after the segmentation process. Further, a segmentation process may be performed after the matting process.

The image integrating unit 15 generates a mask image by integrating the respective processing results of the segmentation processing unit 13 and the matting processing unit 14, and performs a mask process on the original image (step S5). Specifically, in QUADMAP, 255 the pixel values of the foreground region F, the pixel value of the background area B and 0, the segmentation processing region _{U S,} calculated pixel values of the segmentation processing unit 13 (0 or 255) using, for matting processing region U _M, to generate a mask image using the obtained pixel value of the matting unit 14 (0 to 255). Mask processing is performed on the original image using the generated mask image, and a foreground image is cut out from the original image. In the mask processing, pixels having a pixel value of 1 or more in the mask image are cut out from the original image as a foreground image. For a pixel having a pixel value of 254 or less, the transmittance according to the pixel value is set.

  The output unit 16 outputs the cut out foreground image (Step S6).

[Area determination processing]
Next, the area determination processing will be described.

  FIG. 6 is a flowchart illustrating the flow of the area determination process. In the present embodiment, the unknown area U is classified based on the luminance gradient of the original image.

  The area determination unit 12 converts the original image into a luminance image (Step S21), and obtains a differential image by applying a differential filter to the luminance image (Step S22). In the present embodiment, edges are extracted using a Laplacian filter as a differential filter. FIG. 7 shows an example of the differential image.

The area determination unit 12 extracts a pixel whose pixel value is equal to or larger than the threshold value from the differential image obtained in step S22 to obtain a threshold image (step S23). Specifically, the pixel values of all pixels in the same size as the differential image prepared threshold image of 0, the pixel values of the pixels in the threshold image pixel values corresponding to the pixels of the threshold value Th ₀ or more differential image 255. By this processing, only the pixels whose edges appear strongly in the original image are separated. FIG. 8 shows an example of the threshold image. The frame in FIG. 8 is a frame indicating the range of FIGS. 9, 10, and 11 used in the following description.

  The area determination unit 12 divides the threshold image obtained in step S23 into small areas (step S24). FIG. 9 shows an example in which a part of the threshold image of FIG. 8 is divided into small regions. In the present embodiment, the threshold image is divided into 48 × 27 small areas. The size of the small area is not limited to this, and can be set arbitrarily.

Region discriminator 12, for each small region, counts the number of pixels extracted in step S23, to classify the unknown region U in the segmentation process area U _S or matting process area U _M based on the number of pixels (step S25). Specifically, counts the number of pixels of the pixel values in each small area set in the threshold image 255, an unknown region U in the small area where the number of pixels exist threshold Th ₁ or more as the segmentation processing area U _S, The pixel value of the corresponding pixel of TRIMAP is set to 98. Further, as unknown region U matting processing region _{U M} in the small area where the number of pixels is less than the threshold value Th _1, the pixel values of corresponding pixels of TRIMAP to 170. Figure 10 shows an example of classifying the unknown region U segmentation processing region U _S and matting processing region U _M in each small region.

By the above process, comprises a foreground region F and the background region B, QUADMAP classified unknown region U TRIMAP segmentation processing region _{U S} and matting processing region _{U M} is generated.

In the above steps S24 and S25, the threshold image is divided into small regions, and the unknown region U is classified for each small region. However, the unknown region U may be classified for each pixel. Specifically, after the threshold image is obtained in step S23, the number of pixels having a pixel value of 255 in a circle having a radius of 10 pixels centering on the pixels in the unknown area U is counted. For example, the number of white pixels existing in the circle shown in FIG. 11 is counted. Based on the number of pixels counted, to classify the center pixel in the segmentation process area U _S or matting process area U _M. The above processing is performed for all the pixels in the unknown area U. This method can classify the unknown area U more accurately than the area discrimination method of dividing into small areas, but it takes a long processing time because the area is discriminated for each pixel.

[Matting process]
Next, the matting process will be described.

  FIG. 12 is a flowchart illustrating the flow of the matting process.

  The matting processing unit 14 extracts a contour pixel indicating the contour of the foreground area F and the background area B from each of the foreground area F and the background area B (step S41).

  Here, the process of extracting the contour pixels will be described with reference to the flowchart of FIG.

  The matting processing unit 14 scans the QUADMAP to search for a change point at the boundary between the foreground area F and the unknown area U and the boundary between the background area B and the unknown area U (step S411).

  Then, it is determined whether or not the change point is included in the contour line pixel (step S412). Whether or not the change point is included in the contour pixel may be determined based on the distance, angle, interval, and the like with respect to the previously extracted contour pixel. In the present embodiment, the sampling coefficient is set to 0.75. Randomly determined according to the probability.

  FIG. 14 shows an example of pixels at the boundary between the background area B and the unknown area U. FIG. 15 shows an example of a pixel extracted as a contour line pixel of the background area B. The pixels b1 and b2 in FIG. 15 are pixels that have not been extracted as contour pixels.

  As another method of extracting the contour pixels, pixels may be removed from the contour pixels at regular intervals. For example, after scanning the entire QUADMAP to extract contour pixels of the foreground area F and the background area B, pixels are removed from the contour pixels at predetermined intervals. As a result, contour lines can be thinned out at equal intervals, and expectations can be placed on the precision.

Returning to FIG. 12, the matting processing unit 14 selects a candidate point from the contour pixels extracted in step S41 (step S42). Matting processing unit 14, based on the Euclidean distance between the contour pixels extracted by the pixel and step S41 belonging to matting treatment region U _M, the candidate points for color sampling method of the pixel from the contour pixel elect. For example, selects the 10 candidate points from each contour pixel matting processing region U _M contour pixel and the background area of the foreground region F in order close to the pixel belonging to B. The number of candidate points may be a number that can maintain sufficient accuracy for the color sampling method.

  In the present embodiment, the number of distance calculations when selecting a candidate point is reduced by sharing the candidate point between neighboring pixels. Here, sharing of candidate points will be described with reference to the flowchart of FIG.

Matting processor 14 sets the window to matting treatment region _{U M} (step S421). As shown in FIG. 17, in the present embodiment, a square window of 9 × 9 pixels is set.

Matting processing unit 14, the central pixel p window determines whether belonging to the matting treatment region _{U M} (step S421). If the center pixel p does not belong to the matting treatment region _{U M} (NO in step S421), matting processing unit 14, without sharing the candidate points in the window, the pixels other than the center pixel p in the window _{q lm (∈U M, l,} m = 0, ..., 8) determining a candidate point for (step S426).

If the center pixel p belongs to the matting treatment region _{U M} (YES in step S421), determining a candidate point for the central pixel p (step S423). In FIG. 17, ten candidate points fs are obtained from the contour pixels f of the foreground area F, and ten candidate points bs are obtained from the contour pixels b of the background area B.

Matting processing unit 14 determines whether to share the candidate point of the center pixel p and the pixel q _lm in window (step S424). The determination as to whether to share the candidate point can be made based on the pixel values of the center pixel p and the pixel _qlm . In the present embodiment, it is determined that the candidate points are shared in the same window, and the process of step S424 always proceeds to YES.

The matting processing unit 14 shares the candidate point of the pixel _{qlm with} the candidate point of the center pixel p (step S425). In the example of FIG. 17, the candidate point of the pixel q _lm is set to be the same as the candidate point fs, bs of the center pixel p.

For all pixels belonging to the matting treatment region _{U M} to determine the candidate point to repeat the processing from step S422～S426 (step S472).

In the above step S424, determines that compares the pixel values of the pixel q _lm of the central pixel p, when the difference between the pixel values is less than the threshold value ε share a candidate point of center pixel p and the pixel q _lm However, when the pixel value difference is equal to or larger than the threshold value ε, it may be determined that the candidate points are not shared. When the colors of the pixels are similar, it is considered that the transmittance used for the mask is also similar, so that the approximation by sharing the candidate points approaches the correct value. Three channels such as RGB may be used as the pixel value, or the luminance of the pixel may be used.

Returning to FIG. 12, the matting processing unit 14 calculates the α value and the reliability f value for all combinations of the candidate points of the foreground area F and the candidate points of the background area B selected in step S42 (step S43). ). In the present embodiment, three α values having high f values and the f value are averaged to _obtain the initial α value and the reliability f value of the pixel p _ij .

Further, matting processing unit 14, for all the pixels p _ij matting processing region U _M, calculates the average and variance-covariance matrix in nine pixels including the peripheral pixels p _ij, obtains the weight W _ij (step S44).

The matting processing unit 14 updates the α value of the pixel p _ij using the initial α value for a pixel having a high f value and using the weight W _ij for a pixel having a low f value (step S45).

  Note that the processes in steps S43 to S45 described in Non-Patent Document 2 can be used.

Matting processing unit 14 obtains the pixel values of the matting process area _{U M} with the value obtained by multiplying 255 to the α value obtained in step S45 (0 to 1.0) to generate the mask image (step S46).

By the above processing, it is possible to obtain the α value of each pixel of the matting treatment region U _M.

As described above, in the present embodiment, the input unit 11 inputs an original image and a label image in which labels of the foreground area F, the background area B, and the unknown area U are added to the original image, and the area discriminating unit 12 There classifies unknown region U segmentation processing region U _S and matting processing region U _M based on the feature amount obtained from the original image, the segmentation processing unit 13 divides the pixels of the segmentation process area U _S to the foreground and background, matting processor 14 obtains the transmittance of the pixels of the matting treatment region U _M, the image integration unit 15 generates a mask image by integrating the processing result of the segmentation processing unit 13 and the matting unit 14, the mask image Is used to extract a foreground image in which the subject is captured from the original image. According to this embodiment, classifying the unknown region U, the matting treatment region U _M but is refined and segmentation processing region U _S intact sophistication be applied segmentation process of applying the matting process takes time By doing so, the area to which the matting process is applied can be reduced, so that high-speed operation can be achieved while maintaining precision.

According to this embodiment, when matting processor 14 uses the color sampling techniques, by discretely extracted contour pixels of the foreground region F and the background region B, a target pixel matting processing region U _M It is possible to reduce the number of times of calculation of the distance to the contour pixel and increase the speed.

According to this embodiment, when matting processor 14 use a color sampling technique, seeking permeability based on the positional relationship between the candidate points selected from the target pixel and contour pixels matting processing region U _M processing In, by sharing the candidate points for adjacent target pixels, the number of selections of the candidate points can be reduced and the speed can be increased.

DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus 11 ... Input part 12 ... Area discrimination part 13 ... Segmentation processing part 14 ... Matting processing part 15 ... Image integration part 16 ... Output part

Claims (7)

Input means for inputting information that classifies the processing target image and the processing target image into a foreground area, a background area, and an unknown area;
Area determining means for classifying the unknown area into a first area and a second area based on a feature amount obtained from the processing target image;
Segmentation processing means for performing segmentation on the first area, and classifying each pixel of the first area into the foreground area and the background area;
Matting processing means for performing alpha matting on the second area to determine the transmittance of each pixel in the second area;
An image extraction unit that extracts a foreground image from the processing target image using a mask image obtained by integrating the processing result for the foreground region, the background region, and the first region with the processing result for the second region;
An image processing apparatus comprising:   The matting processing means discretely extracts a contour pixel from which a candidate point to be used when calculating the transmittance of each pixel is selected from a contour line of the foreground region and a contour line of the background region. The image processing apparatus according to claim 1, wherein:   3. The image processing apparatus according to claim 1, wherein the matting processing unit shares a candidate point used when calculating the transmittance of each of the pixels between the adjacent pixels. 4. An image processing method executed by the image processing device,
Inputting information that classifies the processing target image and the processing target image into a foreground area, a background area, and an unknown area;
Classifying the unknown area into a first area and a second area based on a feature amount obtained from the processing target image;
Performing a segmentation on the first area to classify each pixel of the first area into the foreground area and the background area;
Performing alpha matting on the second area to determine the transmittance of each pixel in the second area;
Extracting a foreground image from the processing target image using a mask image obtained by integrating the processing result for the foreground region, the background region, and the first region with the processing result for the second region;
An image processing method comprising:   The step of determining the transmittance of each pixel in the second area includes the step of selecting a candidate for selecting a candidate point to be used in determining the transmittance of each pixel by selecting a contour pixel of the foreground area and a contour of the background area. The image processing method according to claim 4, wherein the image is discretely extracted from the image data.   The step of obtaining the transmittance of each pixel in the second area, wherein a candidate point used when obtaining the transmittance of each pixel is shared between adjacent pixels. Image processing method.   An image processing program for causing a computer to operate as each unit of the image processing apparatus according to claim 1.