KR20160069804A - Method for processing image for blur region detection and image processing apparatus thereof - Google Patents

Abstract

A method of extracting an edge characteristic for a pixel using a pixel included in an image and surrounding pixels and extracting the pixel as a non-blur pixel, a defocus blur pixel, and a motion blur Classifying the image into motion blur pixels and dividing the image into a plurality of segments including at least one pixel based on the similarity of colors between adjacent pixels and dividing the number of each classified pixels contained in the plurality of segments Blur area, a defocus blur area, and a motion blur area based on the determination result of the determining step. The non-blur area, the defocus blur area, and the motion blur area are determined as the non-blur area, To the image processing apparatus.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing method for blur area detection,

An image processing method for blur area detection, and an image processing apparatus for performing the same.

Digital images obtained from an image acquisition device such as a digital camera may include various types of blur areas. At this time, it is important to grasp the type and position of the blur area before processing digital images including various types of blur areas. That is, the type and position of the blur area in the digital image obtained from the image acquiring device can be determined by partial deblurring, image enhancement, high-level image segmentation, Object extraction, depth recovery, fast moving object detection and tracking, speed estimation, motion segmentation, down sampling, It is very important to solve problems related to computer vision such as

Accordingly, various techniques for understanding the type and position of the blur area have been developed. On the other hand, the conventional detection of the blur area may be performed as the whole image, or the image processing apparatus may partially detect only one blur area of the defocus blur area or the motion blur area.

Therefore, there is a problem that the image processing apparatus processes an image without consideration of other blur areas not detected in the image.

The present invention provides an image processing method for effectively detecting an image based on a detected blur area, and an image processing apparatus for performing the image processing.

An image processing method according to an embodiment of the present invention includes extracting an edge characteristic for the pixel using a pixel included in an image and surrounding pixels, and calculating the non-blurred pixel using the extracted edge characteristic, blur pixel, a defocus blur pixel, and a motion blur pixel; Dividing the image into a plurality of segments including at least one pixel based on color similarity between adjacent pixels; Determining the plurality of segments as a non-blur area, a defocus blur area, and a motion blur area based on the number of pixels classified in the plurality of segments; And separately processing the determined non-blur area, defocus blur area, and motion blur area.

Wherein the edge characteristic comprises intensity and edge direction consistency of an edge determined by the pixel and surrounding pixels, and wherein classifying the pixel into a non-blur pixel, a defocus blur pixel, and a motion blur pixel comprises: Classifies the pixel into a motion blur pixel if the intensity of the edge and the consistency of the edge direction are large and classifies the pixel into a defocus blur pixel if the intensity of the edge and the consistency of the edge direction are small, , And may classify the pixel into non-blur pixels if the edge direction consistency is small.

On the other hand, the intensity of the edge can be determined based on the magnitude of the gradient in the horizontal direction and the magnitude of the gradient in the vertical direction of the pixel and the surrounding pixels.

The consistency of the edge direction can be determined according to the horizontal gradient distribution of the pixel and the surrounding pixels and the vertical gradient distribution.

On the other hand, the step of determining the non-blur area, the defocus blur area, and the motion blur area based on the number of the classified pixels included in the plurality of segments may include determining one of the plurality of segments According to the number of pixels among the classified Non-Blur pixels, Defocus blur pixels and Motion blur pixels contained in the segment, the one segment is divided into the non-blur area , A defocus blur area, and a motion blur area.

An image processing apparatus according to an embodiment of the present invention includes an edge feature extraction unit for extracting an edge characteristic of the pixel using pixels included in an image and surrounding pixels; A pixel classifier for classifying the pixel into non-blur pixels, defocus blur pixels, and motion blur pixels using the extracted edge characteristics; A segment dividing unit dividing the image into a plurality of segments including at least one pixel based on the similarity of colors between adjacent pixels; A blur area determining unit for determining the plurality of segments as a non-blur area, a defocus blur area, and a motion blur area based on the number of classified pixels contained in the plurality of segments; And an image processing unit for separately processing the determined non-blur area, defocus blur area, and motion blur area.

Here, the edge characteristic includes the intensity of the edge determined by the pixel and the surrounding pixels and the consistency of the edge direction, and the pixel classifier classifies the pixel into a motion blur pixel and a motion blur pixel if the intensity of the edge and the consistency of the edge direction are large. Classifies the pixel into a defocus blur pixel if the intensity of the edge and the consistency of the edge direction are small and if the intensity of the edge is large and the consistency of the edge direction is small, .

Meanwhile, the edge characteristic extraction unit may determine the intensity of an edge for the pixel based on a magnitude of a horizontal gradient and a gradient of the vertical direction of the pixel and surrounding pixels.

In addition, the edge characteristic extraction unit may determine the consistency of the edge direction with respect to the pixel according to the distribution of the gradients in the horizontal direction of the pixel and the surrounding pixels and the distribution of the gradients in the vertical direction.

The blur area determination unit may determine the blur area based on the number of pixels among the classified non-blur pixel, defocus blur pixel, and motion blur pixel included in one segment of the plurality of segments, Blur area, a defocus blur area, and a motion blur area.

A recording medium according to another embodiment of the present invention records a computer program for executing the methods and can be read by a computer.

FIG. 1 is a view for explaining a type of a blur area.
2 is a block diagram illustrating an image processing apparatus according to an embodiment of the present invention.
FIG. 3 is a diagram for explaining a process of determining a type of a blur area for each segment of an image according to an exemplary embodiment of the present invention. Referring to FIG.
4 is a diagram for explaining the process of classifying each pixel based on the relationship between the intensity of an edge and the consistency of an edge direction according to an embodiment of the present invention.
FIG. 5 is a diagram illustrating a relationship between a distribution of a gradient according to an embodiment of the present invention and an Eigenvalue determined in a structural tensor matrix. FIG.
FIG. 6 is a diagram for explaining a process of determining a non-blur area, a motion blur area and a defocus blur area for each segment according to an embodiment of the present invention.
7 is a flowchart illustrating an image processing method based on a detected blurred region for each segment according to an embodiment of the present invention.

In various embodiments of the present invention described herein, an 'image' can be generically referred to as a still image, as well as a moving image such as a video.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, with reference to FIGS. 1 to 7, an image processing method based on segment-detected blur areas according to various embodiments and an image processing apparatus for performing the method are disclosed.

Furthermore, the singular forms "a", "an," and "the" include plural referents unless the context clearly dictates otherwise.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Referring to the accompanying drawings, the same or corresponding elements are denoted by the same reference numerals, and redundant explanations thereof may be omitted.

FIG. 1 is a view for explaining a type of a blur area.

The image can be largely classified into three areas such as a non-blur area, a motion blur area, and a defocus blur area.

Here, the non-blur area means an area where blur (blur) such as motion blur or defocus blur does not exist or is almost not present. The non-blur region may contain strong edges. A strong edge means a strong edge caused by a sudden change in color. The direction of the edges included in the non-blur region is inconsistent, and each edge may have an arbitrary direction.

On the other hand, the motion blur area means a region where a blurred portion such as a motion blur exists. Specifically, the motion blur means that the image acquiring device is shaken at the moment of acquiring an image from an image acquiring device such as a digital camera, or in response to a direction in which the subject moves due to rapid movement (in a direction parallel to the moving direction) And the like.

The pixels included in the motion blur region have strong gradients in a direction perpendicular to the direction in which the object or the image capturing apparatus moves. That is, the motion blur area is a region having a strong edge in a direction parallel to the direction in which the object or the image capturing device moves. The direction of the edges has a specific direction corresponding to the direction of movement (i.e., a direction parallel to the direction of movement) rather than an arbitrary direction, and the edges included in the motion blur region are directionally consistent.

On the other hand, the defocus blur area means a region where a blurred portion such as a defocus blur exists. Specifically, the defocus blur means a blurred portion in which the subject is blurred if the subject is not properly focused at the moment of acquiring the image.

The defocus blur area may include weak edges. Also, the direction of the edges included in the defocus blur area has an arbitrary direction.

On the other hand, when a blur area is detected, consideration is not necessarily given to considering the edges included in the blur area, but considering the edge included in the surrounding area of a certain range adjacent to the relevant area, ) Region, a motion blur region, and a defocus blur region.

Referring to FIG. 1 (a), the image 101 obtained from the image acquiring apparatus includes a dandelion flower as a subject. The image 101 partially includes a defocus blur area 102 and a non-blur area 103. [ The defocus blur area 102 includes a defocus blur, which is a part of a background, not a dandelion flower, which is a subject, but the background part is not focused and the background part is blurred and blurred. In particular, the defocus blur area 102 does not include strong edges because the background part appears in a crushed shape, and the included edges do not have a certain direction but have an arbitrary direction.

On the other hand, the non-blur area 103 is a part of the dandelion flower. Since the dandelion flower was properly focused at the time of acquiring the image, the dandelion flower part is clearly displayed on the image without crushing. Particularly the border of the petals contained in the non-blur area 103. The edges included in the non-blur area 103 have a certain direction, not an arbitrary direction.

1 (b), the image 104 obtained from the image acquiring device includes a person who is skating with the subject. The image 104 partially includes a motion blur area 105 and a non-blur area 106.

The motion blur area 105 is a part of the background except for the person skating. The image acquisition device acquires an image while tracing a person moving on a skate, and thus an edge exists in the background part in the direction in which the image acquisition device moves do.

The motion blur area 105 contains strong edges. The direction of the edges included in the motion blur area 105 has a certain direction in a direction parallel to the motion of the image capturing apparatus.

The non-blur area 106 is a part of a person who is riding on a skate, which is a subject, and acquires an image by using an image acquisition device while closely following a person moving on a skate. Therefore, unlike the motion blur area 105, And the person who moves on the skate is clearly displayed on the image. Particularly, the non-blur area 106 includes a strong edge at the border between the person and the background, and the included edges are edges according to the shape of a person and have a certain direction without having a certain direction.

Referring to FIG. 1 (c), the image 107 obtained from the image acquiring device includes an owl flying to the subject. The image 107 includes a defocus area 108, a motion blur area 109, and a non-blur area 110.

The defocus area 108 is part of the background rather than the owl, and includes a defocus blur that is blurred because the background part is not focused on the owl. In particular, the defocus blur area 108 does not include a strong edge because the background is crushed, and the included edges do not have a certain direction but have an arbitrary direction.

The motion blur area 109 contains the wing portion of the owl. The image is being acquired by the image acquisition device while the owl, which is the subject, is making wings. Therefore, an edge exists in the motion blur area 109 in the direction of the wing.

That is, the motion blur area 109 contains strong edges. The direction of the included edges has a certain direction parallel to the direction in which the wings move.

The non-blur region 110 includes the head portion of the owl. The owl is focused on the subject, so the head of the owl is not blurred and is clearly displayed on the image. The head of the owl also contains strong edges along the shape of the owl, and the included edges have arbitrary orientation.

According to one embodiment of the present invention, the defocus area 108, the motion blur area 109, and the non-blur area 110 partially included in the image 107 as shown in FIG. 1 (c) Can be detected.

2 is a block diagram illustrating an image processing apparatus according to an embodiment of the present invention.

2, an image processing apparatus 200 according to an exemplary embodiment of the present invention includes an edge characteristic extraction unit 210, a pixel classification unit 220, a segment division unit 230, a blur area determination unit 240, And an image processing unit 250.

The edge characteristic extraction unit 210 extracts an edge characteristic of the current pixel using the current pixel and surrounding pixels included in the image. That is, the edge characteristic extracting unit 210 extracts edge characteristics for pixels on a pixel-by-pixel basis. At this time, the edge characteristic for the pixel may include the intensity of the edge and the edge direction consistency determined by the pixel and surrounding pixels.

The edge characteristic extraction unit 210 may determine the intensity of an edge of the current pixel based on the size of the gradient of the horizontal direction of the current pixel and the surrounding pixels and the size of the gradient of the vertical direction. Specifically, the process of determining the strength of an edge of a current pixel by the edge characteristic extraction unit 210 will be described later with reference to FIG.

The edge characteristic extraction unit 210 can determine the consistency of the edge direction with respect to the current pixel according to the distribution of the gradients in the horizontal direction of the current pixel and the surrounding pixels and the distribution of the gradients in the vertical direction. Specifically, the process of determining the edge direction consistency with respect to the current pixel will be described later with reference to FIG.

The pixel classifying unit 220 classifies the current pixel into a non-blur pixel, a defocus blur pixel, and a motion blur pixel using edge characteristics extracted for the current pixel . That is, the pixel classification unit 220 can classify the current pixel according to the intensity of the edge, which is the edge characteristic extracted for the current pixel in the edge characteristic extraction unit 210, and the consistency of the edge direction.

Specifically, the pixel classifier 220 can classify the non-blur pixel, the defocus blur pixel, and the motion blur pixel at once according to the relationship between the intensity of the edge and the consistency of the edge direction. The pixel classifier 220 classifies the current pixel into a motion blur pixel if the intensity of the edge for the current pixel and the consistency of the edge direction are large. Meanwhile, the pixel classifier 220 can classify the current pixel into the defocus blur pixel if the intensity of the edge and the direction of the edge of the current pixel are small. In addition, the pixel classifier 220 can classify the current pixel into a non-blur pixel if the intensity of the edge for the current pixel is large and the consistency in the edge direction is small.

The pixels classified by the pixel classifying unit 220 do not take into account the boundary between the subject and the background included in the image so that the image is divided into an area including at least one motion blur pixel, An area including at least one pixel and an area including at least one non-blur pixel.

The segment division unit 230 divides the image into a plurality of segments. Specifically, the segment dividing unit 230 may divide a plurality of segments including at least one pixel based on similarity of colors between adjacent pixels. That is, the segmented segment in the segment division unit 230 may include one or more adjacent pixels having a similar color.

The segment division unit 230 may divide an image into a plurality of segments so that a boundary portion generated by a sudden change in hue is preserved based on the similarity of pixels adjacent to each other. Meanwhile, the pixel classifier 230 may divide an image into a plurality of super pixels using a super pixel segmentation method.

The blur area determination unit 240 determines a blur area, a defocus blur area, and a motion blur area based on the number of non-blur pixels, defocus pixel blur pixels, and motion blur pixels included in the plurality of segments, . That is, the blur area determining unit 240 determines each of the segments segmented by the segment dividing unit 230 as a non-blur area, a defocus blur area, and a motion blur area.

At this time, the blur area determination unit 240 determines one of the non-blur area, the defocus blur pixel, and the motion blur pixel based on the number of pixels among the non-blur pixel, defocus blur pixel, and motion blur pixel included in one segment of the plurality of segments, A defocus blur area and a motion blur area. The blur area determination unit 240 determines the pixels classified by the pixel classification unit 220 included in each segment according to the type (i.e., whether it is non-blur pixel, defocus blur pixel, and motion blur pixel) Counts the number of pixels of the type, and determines each segment as a non-blur area, a defocus blur area, and a motion blur area based on the pixels of the greatest number of types.

For example, one segment contains six pixels, one of the six pixels is classified as a non-blur pixel, two of the six pixels are classified as motion blur pixels, and three of the six It is assumed that the pixels are classified as defocus blur pixels. At this time, the blur area determining unit 240 may determine the segment as a defocus blur area according to the defocus blur pixel, which is the pixel having the largest number of three pixels from the six pixels included in the segment. On the other hand, when the largest number of types of pixels are of various types, the blur area determining unit 240 determines the current segment as a non-blur area, a defocus blur area, and a motion blur area based on the area determined for the surrounding segments You can decide.

On the other hand, when the current segment can not be determined as the non-blur area, the defocus blur area, and the motion blur area based on the area determined by the neighboring segments, the blur area determining part 240 determines The current segment can be determined as the blur area, the defocus blur area, and the motion blur area according to the determined priority.

The image processing unit 250 may process the non-blur area, the defocus blur area, and the motion blur area determined by the blur area determining unit 240 individually. Specifically, the image processing unit 250 can process the current segment on a segment-by-segment basis according to which region the non-blur region, the defocus blur region, and the motion blur region belong to. More specifically, the image processing unit 250 may perform partial deblurring, image enhancement, high-level image segmentation, object extraction, depth recovery, Processing for fast moving object detection and tracking, speed estimation, motion segmentation and down sampling may be performed according to the type of area determined for the segment Can be performed on a segment basis.

FIG. 3 is a diagram for explaining a process of determining a type of a blur area for each segment of an image according to an exemplary embodiment of the present invention. Referring to FIG.

Referring to FIG. 3, an image 300 is an image acquired by an image acquisition device using an owl as a target object. When the image 300 is inputted, in step 301, the image processing apparatus 200 extracts a feature for the current pixel. At this time, extracting the feature of the current pixel means extracting the characteristics of the edges included in the current pixel and adjacent pixels in a certain range. The edge characteristic for the current block extracted in detail may be consistency of edge strength and edge direction determined by the current pixel and a certain range of neighboring neighboring pixels.

In operation 302, the image processing apparatus 200 extracts characteristics of an edge for each of all pixels included in the image 300, and extracts each pixel as a motion blur pixel, a defocus blur pixel, Non-blur pixel. Therefore, the pixels of the owl part of the target chain in the classified image 310 generated as a result of classification are classified as non-blur pixels, and the pixels near the owl's wings and near the owl and background boundaries are classified into motion blur , And the pixels included in the background can be classified as defocus blur pixels because the focus is not properly adjusted. However, in the classification image 310, the pixels in the vicinity of the boundary between the background and the owl are classified regardless of the boundaries of the owl and the background, and are not preserved in the classification image 310 including the pixel in which the boundary of the owl and the background are classified .

In step 303, the image processing apparatus 200 divides the image 300 into a plurality of segments to determine a blur area while preserving the boundary between the background and the owl. Wherein each segment includes at least one pixel. The criterion by which the segment is divided may be the color similarity between adjacent pixels. A method of dividing a segment based on the similarity of colors between adjacent pixels is a super pixel division method. A segment divided by the super pixel division method is called a super pixel. For example, in the process of splitting into superpixels, P.F. The "Efficient Graph-Based Image Sementation" algorithm of Felzenszwalb can be used. This is a graph-based algorithm. Specifically, each pixel is regarded as one node, and connected to neighboring pixels and edges, and similarity with neighboring pixels is measured from the neighboring pixels, so that similar pixels can be expressed as one area. At this time, one region represented is referred to as a super pixel. The parameters used in the algorithm may be σ, k, min_area, where σ is the σ value of the Gaussian filter, which reduces the artifact of the image. For example, a fixed value of 0.8 is used . On the other hand, k is a parameter indicating how finely divided it is, and the number of super pixels can be adjusted according to the value of k. The smaller the value of k, the finer the partition can be. For example, a small k image may be used for a small image and a large value k may be used for a large image. On the other hand, min_area is a parameter indicating the size of a minimum area that a superpixel can have, and a region smaller than min_area is removed, and the superpixel can be determined by merging with another region.

Meanwhile, the process of dividing an image into super pixels is well known to those skilled in the art, and thus a detailed description thereof will be omitted.

The image processing apparatus 200 divides the image into a plurality of segments by using adjacent pixels having similar colors as one segment based on the similarity of adjacent colors.

On the other hand, since the color suddenly changes near the border of the background and the owl included in the image, if the segment is divided into a plurality of segments based on the similarity of colors, the segment can be divided into a plurality of segments with the boundaries preserved.

In operation 304, the image processing apparatus 200 determines each segment to be one of a motion blur region, a defocus blur region, and a non-blur region using the segmented image 320 and the classified image 310 divided into a plurality of segments . At this time, the image processing apparatus 200 divides each segment into a motion blur area, a defocus blur area, and a defocus blur area based on the type of each pixel (motion blur pixel, defocus blur pixel, non-blur pixel) And a non-blur area.

More specifically, the image processing apparatus 200 refers to the pixels corresponding to the segment of the divided image 320 in the classification image 310, and determines the number of pixels of each type The segment is determined as one of a motion blur area, a defocus blur area, and a non-blur area. In this manner, the type of the region is determined for the segments included in the image, and the type of the region determined for each segment can be stored together with the image in the form of a label.

The output image 330 output through step 304 indicates that the boundary between the owl and the background is preserved and the motion blur area, defocus blur area, and non-blur area are determined.

4 is a diagram for explaining the process of classifying each pixel based on the relationship between the intensity of an edge and the consistency of an edge direction according to an embodiment of the present invention.

Referring to FIG. 4A, the image processing apparatus 200 according to an embodiment of the present invention calculates an edge characteristic for a current pixel based on the intensity of an edge included in a current pixel and surrounding pixels, .

In particular, the strength of an edge with respect to a current pixel is an index indicating whether there is a large amount of change in edge between the current pixel and surrounding pixels. In particular, the larger the change in color between adjacent pixels, the stronger the edge, and the more pixels having a larger color change, the greater the number of edges included in the current pixel and neighboring pixels. Therefore, the magnitude of the gradient The strength of the edge for the current pixel can be determined according to the following equation:

For example, MG (i) can be determined by the following equation (1). Here, MG (i) denotes the MG of the pixel located at the ith position in the current image.

[Equation 1]

In this case, Pi denotes a square patch having a predetermined size, and Npi denotes the total number of pixels included in the patch Pi, and Ix (j ) Denotes the size of the horizontal gradient of the jth pixel in the patch Pi and Iy (j) denotes the size of the vertical gradient of the jth pixel in the patch Pi. k means the order, which is determined experimentally in advance. As the degree becomes larger, the strong edge included in the patch Pi becomes more prominent, and the weak edge is relatively neglected, and MG (i) is determined. That is, when the value of k is large, when the patch Pi contains a strong edge, the value of MG (i) has a relatively larger value, and when the patch Pi contains a weak edge, the value of MG (i) And the noise is reduced.

On the other hand, Ix (j) and Iy (j) may have a normalized value ranging from 0 to 1, where MG (i) j) ^ 2 + Iy (j) ^ 2) can be divided by √2. On the other hand, when the logarithm is obtained by dividing √ (Ix (j) ^ 2 + Iy (j) ^ 2) by √2 and the patch P i includes a strong edge and the patch P i contains a weak edge, ) Can be increased.

On the other hand, the direction of the edge has a direction orthogonal to the gradient. When the size of the gradient is large, the edge included in the patch Pi is a strong edge. If the size of the gradient is small, the edge included in the patch Pi is a weak edge . On the other hand, the consistency of the edge direction with respect to the current pixel is an indicator of how closely the edges included in the current pixel and surrounding pixels are directed. Specifically, the consistency of the edge direction with respect to the current pixel may be a directional coherence (hereinafter, referred to as DC) with respect to the current pixel. The DC for the current pixel can be determined according to the distribution of the horizontal direction gradient of the current pixel and the surrounding pixels and the distribution of the vertical direction gradient. The edge has a direction orthogonal to the gradient. The main direction of the gradient has a direction orthogonal to the main direction of the edge. Thus, the consistency of the principal direction and orientation of the edge can be determined based on the consistency of the main direction and orientation of the gradient. Rather than determining the main direction in particular, it is determined whether there is a principal direction based on directional consistency, and thus it can be determined whether the current pixel is classified as a motion blur pixel.

The DC for the current pixel can be determined using a structural tensor (ST).

On the other hand, the structural tensor matrix ST (i) can be determined by Equation (2). In this case, ST (i) means the ST of the pixel located at i-th position in the current image.

,

(즉, 구조적인 텐서 행렬 ST(i)의 아이젠벨류 디컴포지션(decomposition)에 의해 아이젠 벨류

,

가 결정됨)의 차이가 패치 Pi에서 그레디언트의 이방성(anisotropy)의 정도를 결정할수 있다. 즉, 현재 픽셀 및 주변 픽셀에 의해 결정되는 그래디언트의 분포는 상기 아이젠 벨류

,

의 차이에 의해서 결정될 수 있다. 아이젠 벨류와 그래디언트 분포 사이의 구체적인 관계는 도 5에서 후술하도록 하겠다.In this case, Pi denotes a square patch having a certain size, which is located at the center of the current pixel located at the i-th current image, Npi denotes the total number of pixels included in the patch Pi, Ix (j ) Denotes the size of the horizontal gradient of the jth pixel in the patch Pi and Ix (y) denotes the size of the vertical gradient of the jth pixel in the patch Pi. The relative eigenvalues derived from the structural tensor matrix ST (i)

,

(I. E., By decomposition of the structural tensor matrix ST (i)

,

Is determined) can determine the degree of anisotropy of the gradient in the patch Pi. That is, the distribution of the gradients, which is determined by the current pixel and the surrounding pixels,

,

As shown in FIG. The specific relationship between the distribution and the gradient distribution will be described later with reference to FIG.

On the other hand, the image processing apparatus 200 can calculate DC (i) according to Equation (3). In this case, DC (i) denotes DC of the pixel located at i-th position in the current image.

&Quot; (3) "

,

는 구조적 텐서 행렬 ST(i)의 아이젠 벨류 디컴포지션에 의해 산출된 아이젠 벨류를 의미한다. 현재 픽셀 및 주변 픽셀에 의해 결정되는 그래디언트의 분포는 상기 아이젠 벨류

,

의 차이에 의해서 결정될 수 있다. 한편, DC(i)는 아이젠 벨류

,

의 차이를 제곱함으로써 항상 양의 값을 갖게 된다. 한편 DC(i)를 정규화하기 위해 아이젠 벨류

,

의 차이를

,

의 합으로 나눌 수 있다.At this time

,

Denotes the Eigenvalue calculated by the Eisenvaldie composition of the structural tensor matrix ST (i). The distribution of the gradients, which is determined by the current pixel and the surrounding pixels,

,

As shown in FIG. On the other hand, DC (i)

,

The value of the square is always positive. On the other hand, in order to normalize DC (i)

,

Difference between

,

.

 When the edges included in the patch Pi are aligned in a specific direction, DC (i) is close to 1, and when the edges included in the patch Pi are not aligned and there is no main direction, DC (i)

On the other hand, referring to FIG. 4 (a), the x-axis of the graph represents DC and the y-axis represents MG. The image processing apparatus 200 may classify the current pixel into a non-blur pixel, a motion blur pixel, and a defocus blur pixel according to the magnitude of the DC for the current pixel and the magnitude of the MG.

The image processing apparatus 200 can classify the pixels belonging to the range 410 in which the MG is large and the DC is small as non-blur pixels. The non-blur pixel in which the blur does not exist has a large MG (i) indicating the intensity of the edge with respect to the current pixel because the subject is clearly expressed. On the other hand, the portion is mostly the shape of the subject or background, The direction of the edges included in the current pixel and the surrounding pixels is not aligned in a specific direction, and the value of DC is small.

On the other hand, the image processing apparatus 200 can classify pixels belonging to the range 420 where DC is large and MG is large as motion blur pixels. That is, the motion blur means that the object appears blurry due to the motion of the object or the image acquiring device, and the strengths of the edges included in the current pixel and the surrounding pixels are large and the edges are aligned in a specific direction, .

On the other hand, the image processing apparatus 200 can classify pixels belonging to the range 430 in which the DC is small and the MG is small, as the defocus blur pixel. In other words, the defocus blur means that the subject is blurred in a blurred form because the focus is not focused on the subject, and the intensity of the edges of the current pixel and the surrounding pixels is small and the edges have an arbitrary direction DC and MG are both small.

On the other hand, pixels classified as motion blur pixels have large DC and MG, and defocus blur have a small DC and MG tendency. Pixels classified as non-blur pixels have a large MG and a small DC, which partially includes the tendency of the DC or MG of the motion blur pixel and the focus blur pixel. Therefore, when the image processing apparatus first classifies pixels into non-blur pixels and blur pixels (motion blur pixels and defocus blur pixels), and classifies blur pixels into motion blur pixels and defocus blur pixels, The focus blur pixel has a tendency of completely different MG, DC, while the non-blur pixel has a tendency of some of the motion blur and defocus blur pixel MG, DC, and when classifying non-blur pixel and blur pixel, Some pixels, which may be classified as motion blur pixels and defocus blur pixels, can be classified as non-blur pixels and thus can not be finely classified into non-blur pixels, motion blur pixels and defocus blur pixels.

However, the image processing apparatus 200 according to an embodiment of the present invention first sets the reference point 400 to determine the range of classification into three types of pixels, and starts from the reference point and selects three types A reference line 440 for determining a range to be classified as a pixel of pixels. Accordingly, the image processing apparatus 200 can classify the current pixel into a motion blur pixel, a defocus blur pixel, and a non-blur pixel at a time, and finely divides the pixel into a motion blur pixel, a defocus blur pixel, and a non- Can be classified.

On the other hand, referring to FIG. 4 (b), the image processing apparatus 200 determines the current pixel as a motion blur pixel, a defocus blur pixel, a non-blur pixel . The image processing apparatus 200 may classify all the pixels included in the image into a motion blur pixel, a defocus blur pixel, and a non-blur pixel according to the classification process for the current pixel. Each point shown in FIG. 4B means one pixel included in the image.

FIG. 5 is a diagram illustrating a relationship between a distribution of a gradient according to an embodiment of the present invention and an Eigenvalue determined in a structural tensor matrix. FIG.

Referring to FIG. 5A, the image acquiring apparatus acquires an image including a subject owl and a background. At this time, the image may include a patch 610, a patch 620, and a patch 630. The patches 610, 620, and 630 are square regions used for extracting the edge characteristic, which means a region in which a current pixel is located in the center of a square region and includes a peripheral pixel in a certain range.

Referring to FIG. 5B, the horizontal and vertical gradients of each pixel included in the patch 610 are shown. At this time, the patch 610 includes the head portion of the owl. Since the head portion of the owl included in the patch 610 is focused and has no motion, the image processing apparatus 200 can classify the pixel located in the middle of the patch 610 into a non-blur pixel . At this time, since the edge included in the patch 610 is strong, the patch 610 includes many pixels having a large gradient size. Also, since the edges included in the patches 610 are determined according to the shape of the owl head, the edges have arbitrary directions without a main direction.

는 패치에 포함된 주된 방향의 그래디언트의 상대적인 크기를 나타내고, 아이젠 벨류

는 패치에 포함된 주된 방향에 수직한 방향의 그래디언트의 상대적인 크기를 나타낸다. 패치(610)의 아이젠벨류

,

는 패치에 포함된 픽셀들의 수평방향의 그래디언트 및 수직방향의 그래디언트가 주된 방향 없이 넓게 분포되어 있기 때문에, 모두 큰 값을 가지게 된다. 따라서 아이젠 벨류

와

의 차이는 작기 때문에 DC도 작게 된다.Thus, the distribution of the horizontal and vertical gradients of the pixels included in the patch 610 shows a widespread form. At this time,

Represents the relative size of the gradient in the main direction included in the patch,

Represents the relative size of the gradient in the direction perpendicular to the main direction included in the patch. Eigenvalue of the patch 610

,

Are all of a large value because the horizontal directional gradients and the vertical directional gradients of the pixels included in the patch are widely distributed without the main direction. Therefore,

Wow

The DC is also small.

,

는 작은 값을 가진다. 따라서 아이젠 벨류

와

의 차이는 작기 때문에 DC도 작게 된다.Referring to FIG. 5C, the size of the gradients in the horizontal direction and the vertical direction of each pixel included in the patch 620 is shown. In particular, the background of the patch 620 is in the form of a blurred background because the background is not properly focused. The image processing apparatus 200 may classify the center pixel of the patch 620 as a defocus blur pixel. The patch 620 includes a weak edge, and the horizontal direction of the pixels included in the patch 620 and the size of the vertical gradient are nearly zero. Accordingly, the gradients of the horizontal direction and the vertical direction of the pixels included in the patch 620 are distributed in a very narrow region. Therefore,

,

Has a small value. Therefore,

Wow

The DC is also small.

Referring to FIG. 5D, the size of the gradients in the horizontal direction and the vertical direction of each pixel included in the patch 630 is shown. At this time, the patch 630 is a wing portion of the owl, but the focal point is properly focused on the wing of the owl, but the strong edges included in the patch 630 due to the wing are arranged in a direction parallel to the wing- And the image processing apparatus 200 can classify pixels located in the center of the patch 630 into motion blur pixels.

는 큰 값을 가지고 주된 방향의 수직된 방향의 그래디언트의 상대적인 크기를 나타내는 아이젠 벨류

는 작은 값을 가진다. 따라서 아이젠 벨류

와

의 차이는 크기 때문에 DC도 크게 된다.The patch 630 includes strong edges, and in particular, since the edges included in the patch 630 have a certain directionality, the sizes of the gradients in the horizontal direction and the vertical direction of the pixels included in the patch 630 in an oval shape are distributed do. That is, the distribution form of the gradient exhibits a certain directionality. Thus, the Eisenvalue representing the relative size of the gradient in the main direction

Represents the relative magnitude of the gradient in the vertical direction of the main direction with a large value,

Has a small value. Therefore,

Wow

The DC is also large.

FIG. 6 is a diagram for explaining a process of determining a non-blur area, a motion blur area and a defocus blur area for each segment according to an embodiment of the present invention.

6A is a diagram illustrating an image 610 in which the image processing apparatus 200 classifies pixels into non-blur pixels, motion blur pixels, and defocus blur pixels. At this time, the area included in the owl near the boundary between the object owl and the background and the area included in the background may be classified into the same type of pixel, so that the background and the boundary of the object are preserved near the boundary between the background and the subject Pixels can not be classified.

6 (b), the image processing apparatus 200 generates an image 620 by dividing an image into a plurality of segments based on color similarity. At this time, since each segment is divided based on similarity to color, the image is divided into a plurality of segments so that the boundary between the object owl and the background is preserved.

Referring to FIG. 6C, the image processing apparatus 200 performs a process for dividing each segment, as shown in FIG. 6A, The type of the region of the segment including the corresponding pixels can be determined according to the type of the included pixel. Specifically, the process of determining the non-blur area and the motion blur area defocus blur area for each segment is as follows.

The image processing apparatus 200 may determine l ^* (i) for the i-th segment in the image according to Equation (4). In this case, l * (i) denotes the label (N, D, M) of the region determined for the i-th segment. The non-blur area is labeled N, the defocus blur area is labeled D, and the motion blur area is labeled M.

&Quot; (4) "

argmax {f (x)} means x where the function f (x) is maximized. card (A) means the number of each element of the set A. Meanwhile, Ri, l included in Equation (4) is determined by Equation (5).

&Quot; (5) "

Ri denotes a segmented i-th segment area in the image, L (x) is a function that returns a label of a pixel x, and x is an x-th pixel included in the i-th segment in the image.

Therefore, Ri, l means a set of pixels whose labels of the pixels included in the i-th segment region are l (for example, 1 is one of N, D and M).

The image processing apparatus 200 determines the number of pixels of each type classified according to the type of pixels included in the segment and determines the type of the region for the segment based on the pixels of the type having the largest number of pixels .

7 is a flowchart illustrating an image processing method based on a detected blurred region for each segment according to an embodiment of the present invention.

Referring to FIG. 7, in operation 710, the image processing apparatus 200 extracts an edge characteristic of a pixel using a pixel included in the image and surrounding pixels, and outputs the pixel to the non-blur pixel , Defocus blur pixel, and motion blur pixel. The edge properties for a pixel may include the intensity and edge direction consistency of the edge determined by the pixel and surrounding pixels. The image processing apparatus 200 classifies the pixels into motion blur pixels if the intensity of the edge and the consistency of the edge direction are large. The image processing apparatus 200 classifies the pixels into defocus blur pixels when the intensity of the edge and the consistency of the edge direction are small. The image processing apparatus 200 classifies the pixels into non-blur pixels when the intensity of the edge is large and the consistency in the edge direction is small. The intensity of the edge determined by the pixel and the surrounding pixels can be determined based on the magnitude of the gradient in the horizontal direction of the pixel and the surrounding pixels and the magnitude of the gradient in the vertical direction. On the other hand, the consistency of the edge directions determined by the pixels and the surrounding pixels can be determined according to the gradient distribution in the horizontal direction and the gradient distribution in the vertical direction of the pixel and the surrounding pixels.

In step 720, the image processing apparatus 200 divides the image into a plurality of segments including at least one pixel based on the similarity of colors between adjacent pixels.

In step 730, the image processing apparatus 200 determines the plurality of segments as the non-blur area, the defocus blur area, and the motion blur area based on the number of the classified pixels contained in the plurality of segments. The image processing apparatus 200 may divide one segment into a non-blur area, a non-blur area, and a motion blur area according to a number of pixels among the classified non-blur pixel, defocus blur pixel, and motion blur pixel included in one segment of the plurality of segments. The focus blur area, and the motion blur area.

In operation 740, the image processing apparatus 200 processes the determined non-blur area, defocus blur area, and motion blur area separately.

It is to be understood that the terms "comprises", "comprising", or "having" as used in the foregoing description mean that the constituent element can be implanted unless specifically stated to the contrary, But should be construed as further including other elements.

The above-described embodiments of the present invention can be embodied in a general-purpose digital computer that can be embodied as a program that can be executed by a computer and operates the program using a computer-readable recording medium. The computer-readable recording medium includes a storage medium such as a magnetic storage medium (e.g., ROM, floppy disk, hard disk, etc.), optical reading medium (e.g., CD ROM,

The present invention has been described with reference to the preferred embodiments. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the present invention is defined by the appended claims rather than by the foregoing description, and all differences within the scope of equivalents thereof should be construed as being included in the present invention.

Claims (11)

A method for extracting an edge characteristic for a pixel using a pixel included in an image and surrounding pixels, and using the extracted edge characteristic to convert the pixel into a non-blur pixel, a defocus blur pixel, And a motion blur pixel;
Dividing the image into a plurality of segments including at least one pixel based on color similarity between adjacent pixels;
Determining the plurality of segments as a non-blur area, a defocus blur area, and a motion blur area based on the number of pixels classified in the plurality of segments; And
Processing the determined Non-Blur area, the defocus blur area, and the motion blur area separately.
The method according to claim 1,
The edge characteristic includes intensity and edge direction coherence of the edge determined by the pixel and surrounding pixels,
The step of classifying the pixel into a non-blur pixel, a defocus blur pixel, and a motion blur pixel includes:
Classifying the pixel into a motion blur pixel if the strength of the edge and the consistency of the edge direction are large and classifying the pixel into a defocus blur pixel if the intensity of the edge and the consistency of the edge direction are small, And classifying the pixel into non-blur pixels if the edge direction is less coherent.
3. The method of claim 2,
Wherein the intensity of the edge is determined based on a magnitude of a gradient in a horizontal direction and a magnitude of a gradient in a vertical direction of the pixel and a surrounding pixel.
3. The method of claim 2,
Wherein the consistency of the edge direction is determined according to a gradient distribution in the horizontal direction and a gradient gradient in the vertical direction of the pixel and the surrounding pixels.
The method according to claim 1,
Determining a non-blur area, a defocus blur area, and a motion blur area based on the number of pixels classified in the plurality of segments,
Wherein the one of the plurality of segments includes a plurality of pixels classified into a non-blur pixel, a defocus blur pixel, and a motion blur pixel included in one segment of the plurality of segments, Determining a segment of the motion blur area as one of a non-blur area, a defocus blur area, and a motion blur area.
An edge characteristic extraction unit for extracting an edge characteristic for the pixel using pixels included in the image and surrounding pixels;
A pixel classifier for classifying the pixel into non-blur pixels, defocus blur pixels, and motion blur pixels using the extracted edge characteristics;
A segment dividing unit dividing the image into a plurality of segments including at least one pixel based on the similarity of colors between adjacent pixels;
A blur area determining unit for determining the plurality of segments as a non-blur area, a defocus blur area, and a motion blur area based on the number of classified pixels contained in the plurality of segments; And
And an image processing unit for separately processing the determined non-blur area, defocus blur area, and motion blur area.
The method according to claim 6,
Wherein the edge characteristic comprises intensity and edge direction consistency of an edge determined by the pixel and surrounding pixels,
Wherein the pixel classification unit comprises:
Classifying the pixel into a motion blur pixel if the strength of the edge and the consistency of the edge direction are large and classifying the pixel into a defocus blur pixel if the intensity of the edge and the consistency of the edge direction are small, And classifies the pixel into non-blur pixels if the edge direction is less coherent.
8. The method of claim 7,
Wherein the edge characteristic extracting unit comprises:
And determines an intensity of an edge with respect to the pixel based on a magnitude of a gradient in a horizontal direction and a magnitude of a gradient in a vertical direction of the pixel and a surrounding pixel.
8. The method of claim 7,
Wherein the edge characteristic extracting unit comprises:
And determines the consistency of the edge direction with respect to the pixel according to the distribution of the gradients in the horizontal direction and the distribution of the gradients in the vertical direction of the pixel and the surrounding pixels.
The method according to claim 6,
Wherein the blur area determining unit comprises:
Blur area, a defocus blur area, and a motion blur pixel, according to pixels having a large number of the classified non-blur pixel, defocus blur pixel, and motion blur pixel included in one segment of the plurality of segments, Wherein the motion blur area is determined as one of the motion blur areas.
A computer-readable recording medium storing a computer program for executing the method according to any one of claims 1 to 5.

Images