JP5099438B2 - Image processing apparatus and method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus, method, and program, and more particularly, to an image processing apparatus, image processing method, and program that can correct image blur while maintaining the size of a bright area in an image. .

  Conventionally, when a camera captures a subject, if the subject moves relative to the camera or the camera moves relative to the subject, motion blur occurs in an image captured by the camera. When the camera outputs an image with motion blur, and the image is supplied to the display, the image with motion blur is displayed on the display.

  Therefore, there exists an image processing apparatus that performs image processing for suppressing motion blur that occurs in an image output from a camera. In this image processing apparatus, image processing that suppresses motion blur by calculating a relative motion amount between a camera and a subject based on motion blur that has occurred in an image, and inferring motion blur at the time of imaging from the motion amount. I do.

  For example, in an image in which motion blur occurs, the pixel value gradually changes at the edge portion of the image. The image processing apparatus corrects the pixel value so that the change in the pixel value in the edge portion becomes steep with respect to the image in which the change in the pixel value in the edge portion is gentle.

  By the way, in general, a display such as a CRT (Cathode Ray Tube) that displays an image on a television receiver has a characteristic that the screen brightness is not proportional to the image level and changes exponentially (gamma characteristic). Have Therefore, for example, a camera such as a television station is configured to perform a gamma process for correcting a gamma characteristic of a display on an image obtained as a result of imaging and output an image subjected to the gamma process. . For example, in the ITU-R BT601, ITU-R BT709, or sRGB standard, the gamma value in gamma processing is defined as about γ = 1 / 2.2, assuming that the CRT is a reference display device.

  However, when the image processing apparatus performs image processing that suppresses motion blur for an image that has been subjected to gamma processing in the camera, the position of the edge portion of the image may shift.

  With reference to FIGS. 1 and 2, an image obtained by performing image processing for suppressing motion blur on an image that has been subjected to gamma processing in the camera will be described.

  FIG. 1 is a block diagram illustrating a system including a camera 11, an image processing device 12, and a display 13.

  The camera 11 includes an imaging unit 14 and a gamma processing unit 15.

  The imaging unit 14 images a subject (not shown) and provides the image signal T1 obtained as a result to the gamma processing unit 15. Here, the image signal T1 is a signal composed of luminance information data of an image. Hereinafter, data of luminance information for each pixel constituting the image is referred to as a pixel value. That is, the image signal T1 does not include image color difference information data.

  The gamma processing unit 15 performs gamma processing on the image signal T1 and provides the image signal T2 obtained as a result to the image processing device 12. That is, the gamma processing unit 15 corrects each pixel value constituting the image signal T1 according to the gamma curve shown in the lower part of FIG. 1, and performs image processing on the image signal T2 constituted by each corrected pixel value. Provided to the device 12.

  The image processing device 12 performs image processing for suppressing motion blur on the image signal T2, and provides the display 13 with the image signal T3 obtained as a result.

  The display 13 displays an image corresponding to the image signal T3.

  2 shows an image signal T0 and an image signal T1 picked up by the image pickup unit 14, an image signal T2 subjected to gamma processing by the gamma processing unit 15, and an image signal T3 subjected to image processing by the image processing device 12. Each waveform in a predetermined part of is shown.

  In FIG. 2, the horizontal axis represents the position (pixel position) in one direction on the image, and the vertical axis represents the pixel value (luminance level) of the image.

  For example, assume that the camera 11 captures a scene in which a white subject exists in front of a black background. The image obtained by imaging includes a black portion where the pixel value is 0 (almost 0) according to a black background, an edge portion where the pixel value changes from 0 to 100, and a pixel value according to a white subject. There is a white portion where the value is 100 (approximately 100). Therefore, FIG. 2 shows the vicinity of an edge portion of an image obtained by capturing a scene in which a white subject exists in front of a black background.

  The first diagram from the top in FIG. 2 shows an image signal T0 imaged by the imaging unit 14 when, for example, the subject and the camera 11 are both stationary.

  In the image signal T0, a black portion having a pixel value of 0 exists on the left side of the position A. An edge portion where the pixel value changes sharply from 0 to 100 exists at the position A. A white portion with a pixel value of 100 exists on the right side of the position A. As described above, when the subject and the camera 11 are stationary, the image signal T0 in which the pixel value of the edge portion changes steeply (perpendicular to the black portion and the white portion), that is, motion blur occurs. No image signal T0 is captured.

  On the other hand, the second diagram from the top in FIG. 2 shows an image T1 captured by the imaging unit 14 when the subject and the camera 11 are moving relatively.

  In the image signal T1, a black portion having a pixel value of 0 exists on the left side of the position B. Between position B and position A, there is an edge portion where the pixel value changes from 0 to 100 in an inclined manner from position B to position A. A white portion having a pixel value of 100 exists on the right side of the position A. The position C is the center position of the edge portion of the image signal T1.

  As described above, when the subject and the camera 11 are relatively moving, the image signal T1 in which the pixel value of the edge portion changes in an inclined manner, that is, the image signal T1 in which motion blur occurs is captured.

  The third diagram from the top in FIG. 2 shows an image signal T2 obtained by performing gamma processing by the gamma processing unit 15 on the image T1 in which motion blur has occurred.

  In the image signal T2, a black portion having a pixel value of 0 exists on the left side of the position B. There is an edge portion that has been subjected to gamma processing between position B and position A. A white portion having a pixel value of 100 exists on the right side of the position A. That is, the image signal T2 includes pixel values obtained by correcting the pixel values of the image T1 in which motion blur has occurred according to the gamma curve as shown in FIG. In the image signal T2, the pixel value of the edge portion changes in a curved line so as to rise rapidly in the vicinity of the position B. As described above, when the gamma processing unit 15 performs the gamma processing on the image signal T1 in which the motion blur has occurred, the pixel value of the edge portion changes in a curved shape so as to rise rapidly in the vicinity of the position B. An image signal T2 is obtained.

  The fourth diagram from the top (first from the bottom) in FIG. 2 shows an image signal T3 that has been subjected to image processing by the image processing device 12 with respect to the image signal T2 that has been subjected to gamma processing.

  In the image signal T3, a black portion having a pixel value of 0 exists on the left side of the position B. Between the position B and the position D on the left side of the position A, there is an edge portion where the pixel value changes sharper than the change in the pixel value of the edge portion of the image T1. A white portion having a pixel value of 100 exists on the right side of the position D. Note that the position C ′ is the center position of the edge portion of the image signal T3.

  As described above, when image processing is performed on the image signal T2 subjected to the gamma processing by the image processing device 12, the pixel value changes more sharply than the change in the pixel value at the edge portion of the image signal T1. An image signal T3 having an edge portion is obtained. That is, an image signal T3 in which motion blur is suppressed more than the image signal T1 is obtained.

  However, in the image signal T3, the position C ′ that is the center position of the edge portion is shifted to the left from the position C that is the center position of the edge portion of the image signal T1 imaged by the imaging unit 14. That is, the center position of the edge portion is shifted in the image signal T3 subjected to the image processing by the image processing device 12 with respect to the image signal T2 subjected to the gamma processing.

  Further, when the edge portion is shifted as described above, the white portion is expanded. As a result, the size of the bright area of the imaged subject widens and the appearance is deteriorated.

In view of this, the present inventor has already disclosed an image processing apparatus or the like that can correct the blur of an image while maintaining the size of a bright area and further maintaining the details.
JP 2008-33592 A

  However, in the technique disclosed in Patent Document 1, since there is little luminance information for an object having a deep color, there is a case where the luminance is lost when the gamma at the time of imaging is taken into account, and correction may not be performed.

  The present invention has been made in view of such a situation, and particularly relates to an image processing apparatus, method, and program that can further suppress image deterioration (blurred image) due to imaging blur.

An image processing apparatus according to one aspect of the present invention acquires a luminance signal from an imaging apparatus that outputs a luminance signal of an image obtained as a result of imaging a subject and outputs the luminance signal, and reverses the luminance signal. An inverse gamma processing unit that performs gamma processing, and a degree of gamma processing applied to the luminance signal that has been subjected to the reverse gamma processing by the reverse gamma processing unit is adjusted based on color information of the image, and adjusted. Correction processing means for performing correction processing for correcting blur of the image with respect to the subsequent luminance signal, and the correction processing means is the luminance signal subjected to the inverse gamma processing by the inverse gamma processing means. A gamma characteristic adjusting unit that adjusts the degree of gamma processing based on the color information of the image, and the brightness after the adjustment by the inverse gamma characteristic adjusting unit. Of the pixel values constituting the signal, the pixel value of the target pixel, the pixel value of the pixel spaced in one direction from the target pixel, and the pixel value of the pixel spaced in the other direction from the target pixel are used. A calculation unit that performs a predetermined calculation and a pixel value of the pixel of interest separated from the pixel of interest in one direction based on a calculation result of the calculation unit, or a pixel value of the pixel of interest in the other direction that having a a replacement means for replacing the one of the pixel values of spaced apart pixel.

  The apparatus further includes gamma processing means for performing the same gamma processing as the gamma processing performed in the imaging apparatus on the luminance signal subjected to the correction processing by the correction processing means.

  The correction processing means is a gamma characteristic adjusting means for adjusting the degree of gamma processing on the luminance signal subjected to the reverse gamma processing by the reverse gamma processing means based on the color information of the image; Of the pixel values constituting the luminance signal after adjustment by the inverse gamma characteristic adjusting means, the pixel value of the pixel of interest, the pixel value of a pixel spaced in one direction from the pixel of interest, and other values from the pixel of interest A calculation unit that performs a predetermined calculation using pixel values of pixels separated in a direction, and a pixel value that is separated from the target pixel in one direction based on a calculation result by the calculation unit And a replacement unit that replaces the pixel value with the pixel value of the pixel separated in the other direction from the target pixel.

  An image processing method and program according to one aspect of the present invention are a method and program corresponding to the above-described image processing apparatus according to one aspect of the present invention.

In the image processing device, method, and program according to one aspect of the present invention, the luminance signal is acquired from an imaging device that outputs a luminance signal of an image obtained as a result of imaging a subject by performing gamma processing, and the luminance The signal is subjected to inverse gamma processing. Next, with respect to the luminance signal that has been subjected to the inverse gamma processing, the degree of gamma processing is adjusted based on the color information of the image, and the blur of the image is compared with the luminance signal that has been adjusted. The luminance signal subjected to the inverse gamma processing is adjusted based on the color information of the image, and the luminance after the inverse gamma characteristic adjustment is adjusted. Of the pixel values constituting the luminance signal, the pixel value of the target pixel, the pixel value of the pixel spaced in one direction from the target pixel, and the pixel value of the pixel spaced in the other direction from the target pixel are used. A predetermined calculation is performed, and based on the calculation result, the pixel value of the pixel of interest is a pixel value of a pixel separated in one direction from the pixel of interest, or a pixel separated from the pixel of interest in the other direction of Ru is replaced by the one of the pixel value.

  The image processing apparatus according to one aspect of the present invention described above can be used as, for example, the entire television system or one component thereof. The television system refers to a system composed of one or more AV (Audio and Visual) devices including a television broadcast receiver.

  As described above, according to the present invention, it is possible to realize image processing that suppresses image deterioration (blurred image) caused by imaging blur. In particular, the degree of suppression can be further increased as compared with conventional image processing.

  Here, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings.

  FIG. 3 shows a configuration example of an embodiment of the image processing apparatus 21 to which the present invention is applied.

  In the example of FIG. 3, the image processing device 21 includes an inverse gamma processing unit 22, a correction processing unit 23, and a gamma processing unit 24.

  The inverse gamma processing unit 22 is provided with an image signal T2. The image signal T2 is a signal obtained as a result of the gamma processing unit 15 performing gamma processing on the image signal T1. The image signal T1 is a signal obtained as a result of being imaged by the imaging unit 14 of the camera 11.

  The inverse gamma processing unit 22 performs gamma processing on the image signal T2. Here, as the gamma processing, processing in which the image signal T2 has the same characteristics as the image signal T1 before the gamma processing is performed by the gamma processing unit 15 is employed. When inverse gamma processing is performed on the image signal T2, an image signal T4 is obtained and provided to the correction processing unit 23.

  For example, the inverse gamma processing unit 22 converts the pixel value of the image signal T2 provided from the gamma processing unit 15 into the inverse gamma curve shown on the lower left side of FIG. 3 (that is, the gamma curve shown in FIG. 1). The curve is corrected according to a curve having the opposite characteristics to the above.

  The correction processing unit 23 performs a correction process for correcting image blur on the image signal T4, and provides the image signal T5 obtained as a result to the gamma processing unit 24. For example, the correction processing unit 23 performs a correction process for correcting the pixel value so that the change in the pixel value at the edge portion of the image becomes steep.

  The gamma processing unit 24 performs gamma processing on the image signal T5, and provides the image signal obtained as a result to the display 13. Here, as the gamma processing, the same processing as the gamma processing performed by the gamma processing unit 15 of the camera 11 is employed. For example, the gamma processing unit 24 corrects the pixel value of the image signal T5 according to the gamma curve shown on the lower right side of FIG. 3 (that is, the same curve as the gamma curve shown in FIG. 1).

  Here, in order to facilitate understanding of the present invention, a technique (hereinafter referred to as a conventional technique) disclosed in Patent Document 1 described above will be described with reference to FIGS. As will be described later, the difference between the conventional technique and the present embodiment resides in the configuration of the correction processing unit 23. Therefore, hereinafter, the conventional correction processing unit 23 is referred to as a conventional correction processing unit 23.

  4 shows an image signal T1 picked up by the image pickup unit 14, an image signal T2 subjected to gamma processing by the gamma processing unit 15, an image signal T4 subjected to reverse gamma processing by the reverse gamma processing unit 22, and a conventional technique. The image signal T5 subjected to the correction processing by the correction processing unit 23 is shown.

  The horizontal and vertical axes in FIG. 4 are the same as the horizontal and vertical axes in FIG. That is, the horizontal axis represents the position (pixel position) in one direction on the image, and the vertical axis represents the pixel value (luminance level) of the image.

  In the first diagram from the top in FIG. 4, as in the second diagram from the top in FIG. 2, motion blur occurs as a result of imaging by the imaging unit 14 when the subject and the camera 11 are relatively moving. The image signal T1 is shown. The position C is the center position of the edge portion of the image signal T1.

  The second diagram from the top in FIG. 4 shows the image signal T2 in which the gamma processing unit 15 performs the gamma processing on the image signal T1 in which motion blur has occurred, as in the third from the top in FIG. ing.

  The third diagram from the top in FIG. 4 shows an image signal T4 that has been subjected to inverse gamma processing by the inverse gamma processing unit 22 with respect to the image signal T2 that has been subjected to gamma processing.

  That is, the image signal T4 is the image signal T1 before the gamma processing unit 15 performs the gamma processing. That is, the image signal T4 is an image signal similar to the image signal T1 imaged by the imaging unit 14. In the image signal T4, a black portion having a pixel value of 0 exists on the left side of the position B. Between position B and position A, there is an edge portion where the pixel value changes from 0 to 100 in an inclined manner from position B to position A. A white portion having a pixel value of 100 exists on the right side of the position A.

  The fourth diagram from the top (first from the bottom) in FIG. 4 shows an image signal T5 that has been subjected to correction processing by the conventional correction processing unit 23 for the image signal T4 that has been subjected to inverse gamma processing. Yes.

  In the image signal T5, a black portion having a pixel value of 0 exists to the left of the position E on the right side of the position B. Between the position E and the position F on the left side of the position A, there is an edge portion where the pixel value changes sharper than the change in the pixel value of the edge portion of the image T4. A white portion having a pixel value of 100 exists to the right of the position F. Note that the center position of the edge portion of the image signal T5 is the same position C as the center position of the edge portion of the image T1.

  Next, with reference to FIG. 5, a correction process performed by the conventional correction processing unit 23 on the image signal T4 will be described.

  In FIG. 5, the horizontal axis represents the position (pixel position) in one direction on the image, and the vertical axis represents the pixel value (luminance level) of the image.

  An image signal T4 is shown in the upper part of FIG. The image signal T4 is provided from the inverse gamma processing unit 22 to the conventional correction processing unit 23. In the lower part of FIG. 5, an image signal T5 that has been subjected to correction processing by the conventional correction processing unit 23 is shown. Further, a plurality of points on the image signal T4 or the image signal T5 represent pixel values constituting the image signal T4 or the image signal T5.

  The conventional correction processing unit 23 sets a pixel to be noted as a correction target as a target pixel. Then, the conventional correction processing unit 23 determines a pixel value of the target pixel, a pixel value of a pixel (hereinafter, appropriately referred to as a right pixel) spaced a predetermined number from the target pixel, and a predetermined value from the target pixel. The pixel values of the pixels separated in the left direction by the number (hereinafter, referred to as the left pixel as appropriate) are acquired.

For example, as shown in the upper part of FIG. 5, the conventional correction processing unit 23 includes a pixel value S _n of the target pixel, a pixel value S _{n−1 of} a right pixel that is three rightward away from the target pixel, Three pixel values, ie, a pixel value S _{n + 1 of} the left pixel separated by three to the left from the target pixel, are acquired.

Conventional correction processing unit 23, the pixel value S _n of the pixel of interest of the right pixel pixel value S _n-1, and the leftward pixel on the image signal T4 in Figure 5 representing each pixel value S _{n + 1} of The secondary differential value A at the point representing the pixel value S _n of the target pixel of the line connecting the points in that order is obtained by calculating the following equation (1).

A = 2 × S _n −S _n−1 −S _{n + 1} (1)

Further, the conventional correction processing unit 23 obtains the difference value B between the pixel value S _n−1 of the right pixel and the pixel value S _{n + 1} of the left pixel by calculating the following equation (2). To do. The difference value B represents the slope of a straight line connecting the points on FIG. 5 representing the pixel value S _n−1 of the right pixel and the pixel value S _{n + 1} of the left pixel.

B = S _n-1 −S _{n + 1} (2)

Then, the conventional correction processing unit 23 determines whether or not the secondary differential value A obtained as a result of calculating Expression (1) is greater than 0, and the difference value B obtained as a result of calculating Expression (2) is Based on whether or not the pixel value is greater than 0, the pixel value S _n of the target pixel is set to one of the pixel value S _{n−1 of the} right pixel and the pixel value S _{n + 1} of the left pixel. replace.

For example, for the pixel value S _{n of} the target pixel, the pixel value S _{n-1 of} the right pixel, and the pixel value S _{n + 1} of the left pixel shown in the upper part of FIG. The difference value B is also greater than 0. In this way, when the secondary differential value A is greater than 0 and the difference value B is greater than 0, the conventional correction processing unit 23 converts the pixel value _Sn of the target pixel to the right pixel. To the pixel value of the pixel value S _n−1 .

Note that the conventional correction processing unit 23 uses the pixel value S _n of the target pixel as the pixel value S of the right pixel even when the secondary differential value A is 0 or less and the difference value B is 0 or less. _{Replace with n-1} . Further, when the secondary differential value A is greater than 0 and the difference value B is 0 or less, the conventional correction processing unit 23 uses the pixel value _Sn of the target pixel as the pixel value of the left pixel. Replace with the pixel value of Sn _{+ 1} . Conventional correction processing unit 23 becomes a second-order differential value A is 0 or less, and, in the case where the difference value B is larger than 0 even, the pixel value S _n of the pixel of interest, the pixel value S _n of left pixels _{Replace with} a pixel value of ₊₁ .

  As described above, the correction processing unit 23 sequentially sets each pixel constituting the image signal T4 shown in the upper part of FIG. 5 as a target pixel and replaces the pixel value of the target pixel, thereby lowering the lower part of FIG. The image signal T5 shown in FIG.

  FIG. 6 is a block diagram illustrating an example of a functional configuration of the conventional correction processing unit 23.

  In FIG. 6, the conventional correction processing unit 23 includes operation units 32 </ b> A and 32 </ b> B and a replacement unit 33.

In the conventional correction processing unit 23, the pixel value of the image signal T4 subjected to the inverse gamma processing from the inverse gamma processing unit 22 is, for example, one pixel in order from the pixel arranged on the right side in FIG. Provided in minutes. In the following, the process executed by the conventional correction processing unit 23 at the timing when the pixel value provided from the inverse gamma processing unit 22 to the conventional correction processing unit 23 is the pixel value S _{n + 1} of the left pixel. Will be described.

  As described with reference to FIG. 5, the calculation unit 32 </ b> A calculates the secondary differential value A according to the equation (1). The calculation unit 32A generates a control signal D2 indicating whether or not the secondary differential value A is greater than 0, and provides it to the replacement unit 33. In other words, the calculation unit 32A includes delay processing units 41 and 42, a multiplication unit 43, subtraction units 44 and 45, and a determination processing unit 46.

  The delay processing units 41 and 42 delay and output the pixel values supplied thereto.

For example, as the image signal T4 from the inverse gamma processing unit 22, the pixel value S _n-1 of the right pixel, the pixel value S _{n of} the target pixel, and the pixel value S _{n + 1 of the} left pixel are sequentially sequentially arranged in that order. Suppose that it is provided. In this case, as shown in FIG. 6, at the timing when the pixel value S _{n + 1} of the left pixel is input as the image signal T4 from the inverse gamma processing unit 22, the pixel value S of the target pixel is output from the delay processing unit 41. _n is output, and the pixel value S _{n−1 of the} right pixel is output from the delay processing unit 42.

The multiplication unit 43, the pixel value S _n of the pixel of interest from the delay processing section 41 is provided. Multiplier 43 outputs the pixel value S _n of the pixel of interest is multiplied by 2, which is set in advance, a value obtained as a result of multiplying 2 to the pixel value S _n of the pixel of interest (2 × S _n) To do.

The subtractor 44 is provided with the pixel value S _{n + 1 of the} left pixel from the inverse gamma processor 22 and the value (2 × S _n ) from the multiplier 46. Therefore, the subtractor 44 subtracts the pixel value S _{n + 1} of the left pixel from the value (2 × S _n ), and subtracts the value (2 × S _n −S _{n + 1} ) obtained as a result. To supply.

The subtraction unit 45 is provided with the pixel value S _{n−1 of the} right pixel from the delay processing unit 42 and the value (2 × S _n −S _{n + 1} ) from the subtraction unit 44. Accordingly, the subtraction unit 45, the value _{(2 × S n -S n +} 1) from, subtracts the pixel value S _n-1 of the right pixel, the resulting value _{(2 × S n -S n +} 1 -S _n-1 ), that is, the secondary differential value A that is the calculation result of the expression (1) is provided to the determination processing unit 46.

  In this way, the determination processing unit 46 is provided with the secondary differential value A that is the calculation result of the expression (1). The determination processing unit 46 determines whether the secondary differential value A is greater than 0, generates a control signal D2 based on the determination result, and provides the control signal D2 to the replacement unit 33.

  Specifically, for example, if the determination processing unit 46 determines that the secondary differential value A is greater than 0, “1” indicating that the secondary differential value A is greater than 0 as the control signal D2. Is output. On the other hand, when determining that the secondary differential value A is 0 or less, the determination processing unit 46 outputs “0” indicating that the secondary differential value A is 0 or less as the control signal D2.

  The calculation unit 32B calculates the difference value B according to the equation (2). The calculation unit 32B generates a control signal D1 indicating whether or not the difference value B is greater than 0 and provides the control signal D1 to the replacement unit 33. That is, the calculation unit 32B includes a subtraction unit 51 and a determination processing unit 52.

The subtraction unit 51 is provided with the pixel value S _{n + 1 of the} left pixel from the inverse gamma processing unit 22 and the pixel value S _{n−1 of the} right pixel from the delay processing unit 42. The subtraction unit 51 subtracts the pixel value S _{n + 1} from the pixel value S _n−1 and obtains a value (S _n−1 −S _{n + 1} ) obtained as a result, that is, the difference that is the calculation result of the expression (2). The value B is provided to the determination processing unit 52.

  In this manner, the determination processing unit 52 is provided with the difference value B that is the calculation result of Expression (2). The determination processing unit 52 determines whether or not the difference value B is greater than 0, generates a control signal D1 based on the determination result, and provides the control signal D1 to the replacement unit 33.

  When determining that the difference value B is greater than 0, the determination processing unit 52 outputs “1” indicating that the difference value B is greater than 0 as the control signal D1. On the other hand, when it is determined that the difference value B is 0 or less, the determination processing unit 52 outputs “0” indicating that the difference value B is 0 or less as the control signal D1.

Based on the control signals D1 and D2 from the calculation unit 32, the replacement unit 33 sets the pixel value _Sn of the target pixel to the pixel value _{Sn + 1} of the left pixel or the pixel value _Sn-1 of the right pixel. Replace with one of them and output. In other words, the replacement unit 33 includes delay processing units 61 and 62 and selectors 63 to 65.

The delay processing units 61 and 62 delay the pixel values supplied thereto and output them. For example, as the image signal T4 from the inverse gamma processing unit 22, the pixel value S _n-1 of the right pixel, the pixel value S _{n of} the target pixel, and the pixel value S _{n + 1 of the} left pixel are sequentially sequentially arranged in that order. Suppose that it is provided. In this case, as shown in FIG. 6, at the timing when the pixel value S _{n + 1} of the left pixel is input to the correction processing unit 23, the pixel value S _{n-1 of the} right pixel is output from the delay processing unit 62. Is done.

The pixel value S _{n + 1 of the} left pixel is supplied from the inverse gamma processing unit 22 to one input terminal of the selector 63, and the pixel of the right pixel from the delay processing unit 62 is supplied to the other input terminal of the selector 63. The value S _n-1 is supplied. The selector 63 is supplied with a control signal D1 from the determination processing unit 52. The selector 63 outputs one of the pixel value S _{n + 1} of the left pixel and the pixel value S _n-1 of the right pixel in accordance with the control signal D1 from the determination processing unit 52.

For example, when the control signal D1 from the determination processing unit 52 is “1” indicating that the difference value B, which is the calculation result of Expression (2), is greater than 0, the selector 63 receives the signal from the delay processing unit 62. The pixel value S _n−1 of the right pixel of is output. On the other hand, when the control signal D1 from the determination processing unit 52 is “0” indicating that the difference value B, which is the calculation result of Expression (2), is 0 or less, the selector 63 receives the signal from the inverse gamma processing unit 22. The pixel value S _{n + 1} of the left pixel is output.

Further, the pixel value S _{n + 1} of the left pixel from the inverse gamma processing unit 22 is also provided to _one input terminal of the selector 64, and the right value from the delay processing unit 62 is also provided to the other input terminal of the selector 54. A pixel value S _n-1 of the second pixel is provided. The selector 64 is also provided with the control signal D1 from the determination processing unit 52. However, in contrast to the selector 63, the selector 64 outputs the pixel value S _{n + 1} of the left pixel from the inverse gamma processing unit 22 when the control signal D1 from the determination processing unit 52 is “1”. The selector 64 outputs the pixel value S _n−1 of the right pixel from the delay processing unit 62 when the control signal D1 from the determination processing unit 52 is “0”.

  The output from the selector 63 is input to one input terminal of the selector 65, and the output from the selector 64 is input to the other input terminal of the selector 65.

That is, when the control signal D1 from the determination processing unit 52 is “1”, the pixel value S _n−1 of the right pixel from the selector 63 is provided to _one input terminal of the selector 65, and the selector 65 The other input terminal 65 is provided with the pixel value S _{n + 1} of the left pixel from the selector 64. On the other hand, when the control signal D1 from the determination processing unit 52 is “0”, the pixel value _{Sn + 1} of the left pixel from the selector 63 is provided to _one input terminal of the selector 65, and the selector 65 The other input terminal of 65 is provided with the pixel value S _n−1 of the right pixel from the selector 64.

  The selector 65 is provided with a control signal D2 from the determination processing unit 46. Therefore, the selector 65 outputs the pixel value input to one of the two input terminals of the selector 63 in accordance with the control signal D2 from the determination processing unit 46.

  For example, when the control signal D2 from the determination processing unit 46 is “1” indicating that the secondary differential value A, which is the calculation result of the expression (1), is greater than 0, the selector 65 sets the selector 63 to the selector 63. The pixel value input to one connected input terminal is output. On the other hand, when the control signal D2 from the determination processing unit 46 is “0” indicating that the secondary differential value A, which is the calculation result of Expression (1), is 0 or less, the selector 65 is connected to the selector 64. The pixel value input to the other input terminal is output.

That is, the pixel value S _{n−1 of} the right pixel or the pixel value S _{n + 1 of the} left pixel is provided to the input terminal of the selector 65 in accordance with the control signal D1 from the determination processing unit 52. In accordance with the control signal D2 from the determination processing unit 46, one of the pixel value S _{n−1 of} the right pixel and the pixel value S _{n + 1} of the left pixel is output.

Specifically, the control signal D1 from the determination processing unit 52 is “1” indicating that the difference value B is greater than 0, and the control signal D2 from the determination processing unit 46 is the second derivative. When the value A is “1” indicating that it is greater than 0, the selector 65 outputs the pixel value S _n−1 of the right pixel. Further, the control signal D1 from the determination processing unit 52 is “0” indicating that the difference value B is 0 or less, and the control signal D2 from the determination processing unit 46 has a secondary differential value A of 0. In the case of “1” indicating that it is larger, the selector 65 outputs the pixel value S _{n + 1} of the left pixel.

Further, the control signal D1 from the determination processing unit 52 is “1” indicating that the difference value B is greater than 0, and the control signal D2 from the determination processing unit 46 is equal to the secondary differential value A. When it is “0” indicating that it is 0 or less, the selector 65 outputs the pixel value S _{n + 1} of the right pixel. In addition, the control signal D1 output from the determination processing unit 52 is “0” indicating that the difference value B is 0 or less, and the control signal D2 from the determination processing unit 46 is the secondary differential value A. When the value is “0” indicating that is less than or equal to 0, the selector 65 outputs the pixel value S _n−1 of the right pixel.

In this way, in the replacement unit 33 of the conventional correction processing unit 23, the pixel value S _n of the target pixel is changed to the pixel value S _{n + 1 of} the left pixel based on the calculation results by the calculation units 32A and 32B. The pixel value S _n−1 of the right pixel is replaced with _one of the pixel values and output.

Here, an example of image processing executed by the image processing apparatus 21 in FIG. 3 will be described with reference to FIG.
The inverse gamma processing unit 22 of the image processing device 21 is provided with the image signal T2 from the gamma processing unit 15 of the camera 11 (for example, the image signal T2 in FIG. 4). Therefore, in step S11, the inverse gamma processing unit 22 performs inverse gamma processing on the image signal T2. The resulting image signal T4 (for example, the image signal T4 in FIG. 4) is provided to the correction processing unit 23. Thereby, a process progresses to step S12.

  In step S12, the correction processing unit 23 performs the correction process described with reference to FIG. 5 on the image signal T4, and the image signal T5 obtained as a result (for example, the image signal T5 in FIG. 4). Is provided to the gamma processing unit 24. Thereby, a process progresses to step S13.

  In step S <b> 13, the gamma processing unit 24 performs gamma processing on the image signal T <b> 5 and provides the image signal obtained as a result to the display 13. Thereby, the image processing of the example of FIG. 7 is completed.

  As described above, in the image processing device 21, the correction processing unit 23 performs correction processing on the image signal T4 that has been subjected to reverse gamma processing by the reverse gamma processing unit 22. Therefore, the center position of the edge portion of the corrected image signal T5 (for example, the image signal T5 in FIG. 4) is the image signal T1 (for example, the image signal T1 in FIG. 4) captured by the imaging unit 14. It becomes possible to make the change in the pixel value of the edge portion steep without deviating from the center position of the edge portion. That is, it becomes possible to suppress the motion blur of the image.

  By the way, in the above-described example, the inverse gamma processing of the inverse gamma processing unit 22 means that the pixel value of the image signal T2 is represented by the inverse gamma curve shown in the lower left side of FIG. This is a correction process according to a curve having a characteristic opposite to that of the gamma curve.

However, this inverse gamma curve is a curve of y = x ^2.2 as shown in FIG. That is, it can be understood that the inverse gamma processing in this case is equivalent to the processing of raising the pixel value to the power of 2.2.

  Therefore, the conventional correction processing unit 23 in FIG. 6 calculates the above equation (1) using the pixel value raised to the power of 2.2 by the inverse gamma processing unit 22. That is, in this case, the conventional correction processing unit 23 of FIG. 6 substantially calculates the following equation (3).

^{A = 2 × S2 2.2 -S1 2.2} -S3 2.2 ··· (3)

  Here, each of S1, S2, and S3 is the pixel value of the right pixel, the pixel value of the pixel of interest, and the pixel of interest of the image signal T2 output from the camera 11 that are three right away from the pixel of interest. This represents the pixel value of the left pixel that is three pixels away from the pixel in the left direction.

In order to realize the calculation processing of the equation (3), it is necessary to mount a memory having a large storage capacity in the conventional correction processing unit 23. Therefore, when it is necessary to reduce the storage capacity of the memory used by the correction processing unit 23 in the calculation, an approximate curve of an inverse gamma curve, for example, a curve of y = x ² may be employed in the inverse gamma processing. That is, the inverse gamma processing in this case can be grasped as a process of squaring the pixel value.

  In this case, the inverse gamma processing unit 22 may be configured as shown in FIG. That is, in the example of FIG. 9, the inverse gamma processing unit 22 is configured to include a multiplication unit 71. Image signals T <b> 2 from the camera 11 are provided to the two input terminals of the multiplier 71. Therefore, the multiplier 71 multiplies the image signal T2 and the image signal T2, that is, calculates the square of the image signal T2, and provides the image signal T4 obtained as a result to the correction processor 23. Note that the correction processing unit 23 in FIG. 9 has the same configuration as the correction processing unit 23 in FIG.

  In this case, the calculation unit 32A of the conventional correction processing unit 23 substantially calculates the following equation (4) when calculating the above equation (1).

^{A = 2 × S2 2 -S1 2} -S3 2 ··· (4)

  The calculation process of the expression (4) can be realized with a memory having a smaller storage capacity than the calculation process of the expression (3). Accordingly, the storage capacity of the memory used by the conventional correction processing unit 23 for calculation can be reduced.

  The prior art, particularly the conventional correction processing unit 23 has been described above with reference to FIGS.

  By the way, in the dark component of the image signal, luminance information may be lost by the inverse gamma processing of the inverse gamma processing unit 22. Therefore, the present inventor uses color information (color difference signals and R, G, B signals) in the image signal as a correction method applicable to the correction processing unit 23 subsequent to the inverse gamma processing unit 22, After adjusting the degree of gamma processing, in other words, the invention further invented a method of correcting the luminance signal while adaptively controlling the ratio considering the gamma characteristic. By applying such a method, it is possible to perform correction so that luminance information is not lost in the color components. Hereinafter, this method is referred to as the method of the present invention.

  FIG. 10 is a block diagram showing an example of a functional configuration of the correction processing unit 23 realized as a result of applying the method of the present invention to the conventional correction processing unit 23 of FIG.

  The correction processing unit 23 in the example of FIG. 10 includes operation units 32A and 32B, a replacement unit 33, and a gamma characteristic adjustment unit 34.

  The arithmetic units 32A and 32B and the replacement unit 33 have basically the same functions and configurations as the corresponding functional blocks in the example of FIG. Therefore, descriptions of the calculation units 32A and 32B and the replacement unit 33 are omitted.

  The gamma characteristic adjustment unit 34 includes a gain unit 81, a subtraction unit 82, a multiplication unit 83, and an addition unit 84.

  The gain unit 81 is provided with the image signal C0 from the camera 11. Here, the image signal C0 is a signal composed of data of color difference information of an image.

  As shown in FIG. 11, the gain unit 81 is configured to include a subtraction unit 91, a fixed value output unit 92, an absolute value calculation unit 93, and a function unit 94.

  The subtraction unit 91 subtracts the fixed value from the fixed value output unit 92 from the value of the image signal C0, and provides the signal C1 obtained as a result to the absolute value calculation unit 93. Here, the fixed value from the fixed value output unit 82 is an achromatic value. For example, 512 is output from the fixed value output unit 82 for a 10-bit signal.

  The absolute value calculation unit 83 converts the value of the signal C1 into an absolute value and provides the signal C2 obtained as a result to the function unit 84. Here, the signal level of the signal C2 indicates the distance from the achromatic color. Therefore, hereinafter, the value (signal level) of the signal C2 is referred to as chromaticity.

  The function unit 84 holds a function f (α) as shown in FIG. 12, for example. Therefore, the function unit 84 substitutes the value of the signal C2, that is, the chromaticity as the input value α into the function f (α), and determines the output value f1 (C2) as the gain (GAIN). A signal C3 having the gain (GAIN) as a signal level is output from the gain unit 81 and provided to the multiplication unit 83 in FIG.

  On the other hand, a signal T0 composed of image luminance information data (pixel values) and a signal T4 obtained by performing inverse gamma processing on the signal T0 are provided to the subtractor 82. The signal T0 is output from the camera 11, while the signal T4 is output from the inverse gamma processing unit 22. The subtracting unit 82 subtracts the signal T4 from the signal T0 and provides a signal (T0-T4) obtained as a result to the multiplying unit 83 as the output signal T42.

Here, the value (predetermined pixel value) of the signal T0 is described as a value S. Then, the value of the signal T4 becomes the value ^S2.2 . Therefore, the value of the output signal T42 of the subtracting unit 82 is the value (S−S ^2.2 ).

The multiplication unit 83 multiplies the output signal T42 of the subtraction unit 82 by the output signal C3 of the gain unit 81, and provides a signal (T42 × C3) obtained as a result to the addition unit 84 as an output signal T43. Here, the value (signal level) of the output signal C3 of the gain unit 81 is GAIN in FIG. Therefore, the value of the output signal T4 of the multiplication unit 83 is the value {(S−S ^2.2 ) × GAIN}.

In addition to the output signal T43 from the multiplication unit 83, the addition unit 84 is also provided with the output signal T4 from the inverse gamma processing unit 22. The adder 84 adds the signal T4 and the signal T43, and outputs a signal T44 obtained as a result. Here, the value of the signal T4 is a value S ^2.2 as described above. Therefore, the value of the output signal T44 of the adder 84 is the value {(S−S ^2.2 ) × GAIN + S ^2.2 }. The output signal T44 of the adder 84 is provided to the calculators 32A and 32B, respectively.

That is, in the conventional correction processing unit 23 of FIG. 6, the operation unit 32A, the 32B, the pixel value S1 ^2.2 as the pixel value S _n-1 of the right pixel, the pixel value as the pixel value S _n of the target-th pixel S2 ^The pixel value S3 ^2.2 is sequentially provided in the order of ^2.2 and the pixel value S _{n + 1} of the left pixel. Note that, as described above, each of S1, S2, and S3 represents the pixel value of the right pixel and the pixel value of the target pixel that are separated by three in the right direction from the target pixel in the image signal T2 output from the camera 11. , And the pixel value of the left pixel separated by three to the left from the target pixel.

On the other hand, in the correction processing unit 23 of FIG. 10 to which the present invention is applied, the pixel values {(S1−S1 ^2.2 ) × GAIN + S1 as the pixel value S _n−1 of the right pixel are input to the calculation units 32A and 32B. ^2.2} is, the pixel value as the pixel value S _n of the target-th pixel ^{{(S2-S2 2.2) ×} GAIN + S2 2.2} is, and the pixel value as the pixel value S _{n + 1} of the leftward pixel {(S3-S3 ^2.2) × GAIN + S3 ^2.2 } are provided sequentially in that order. As a result, the dark component of the image is equivalent to no inverse gamma correction, and correction can be performed without losing luminance information.

  FIG. 13 is a block diagram showing an example of a functional configuration of the correction processing unit 23 realized as a result of applying the method of the present invention to the conventional correction processing unit 23 of FIG. 9 described above. The correction processing unit 23 in FIG. 13 has basically the same function and configuration as the correction processing unit 23 in FIG.

In the example of FIG. 13, the value of the signal T4 is a value S ^2. Therefore, in the correction processing unit 23 of FIG. 13 to which the present invention is applied, the calculation units 32A and 32B have the pixel value {(S1-S1 ² ) × GAIN + S1 ² } as the pixel value S _n-1 of the right pixel. , the pixel value as the pixel value S _n of the target-th pixel ^{{(S2-S2 2) ×} GAIN + S2 2} is, and the pixel value as the pixel value S _{n + 1} of the leftward pixel ^{{(S3-S3 2) ×} GAIN + S3 2} Are provided sequentially in that order. As a result, the dark component of the image is equivalent to no inverse gamma correction, and correction can be performed without losing luminance information.

Note that the GAIN determination method is not particularly limited to the method of determining using the function of the example of FIG. For example, although not shown, the gamma characteristic adjusting unit 34 may be configured such that the value of the output signal T44 of the gamma characteristic adjusting unit 34 is {( ^S2.2− S × GAIN + ^S2.2 }). 12 is adopted as a function g (α) different from the function in the example of Fig. 12. This function g (α) is used until the chromaticity as an input value reaches a threshold value. The output value is a fixed value, and after the threshold value is exceeded, the function is such that the output value decreases monotonously, and the luminance value Y ′ is calculated from the R, G, and B values, and the luminance A determination method of determining GAIN from the ratio between the value Y ′ and the luminance value Y can be employed.

  As described above, by adopting the conventional technique of Patent Document 1, it is possible to correct blur at the time of shooting while correcting the luminance in consideration of the gamma characteristic at the time of imaging. However, in the conventional technique of this Patent Document 1, since there is little luminance information in a dark region of the image, there is a case where the luminance is lost when the gamma at the time of photographing is taken into consideration and correction is not applied. . Therefore, by applying the method of the present invention, that is, by using the color information of the image (color difference information and R, G, B information), adaptively controlling the ratio considering the gamma characteristic, Correction can be applied to a dark region.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, the programs that make up the software execute various functions by installing a computer embedded in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  FIG. 14 is a block diagram illustrating an example of the configuration of a computer that executes the above-described series of processing using a program. A CPU (Central Processing Unit) 101 executes various processes according to a program stored in a ROM (Read Only Memory) 102 or a storage unit 108. A RAM (Random Access Memory) 103 appropriately stores programs executed by the CPU 101 and data. These CPU 101, ROM 102, and RAM 103 are connected to each other by a bus 104.

  An input / output interface 105 is also connected to the CPU 101 via the bus 104. Connected to the input / output interface 105 are an input unit 106 made up of a keyboard, mouse, microphone, and the like, and an output unit 107 made up of a display, a speaker, and the like. The CPU 101 executes various processes in response to commands input from the input unit 106. Then, the CPU 101 outputs the processing result to the output unit 107.

  The storage unit 108 connected to the input / output interface 105 includes, for example, a hard disk, and stores programs executed by the CPU 101 and various data. The communication unit 109 communicates with an external device via a network such as the Internet or a local area network.

  A program may be acquired via the communication unit 109 and stored in the storage unit 108.

  The drive 110 connected to the input / output interface 105 drives a removable medium 111 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and drives programs and data recorded there. Get etc. The acquired program and data are transferred to and stored in the storage unit 108 as necessary.

  Note that processing other than the processing for correcting the pixel value so that the change in the pixel value at the edge portion of the image becomes steep can be applied to the processing for correcting the blur of the image.

  Note that the processes described with reference to the flowcharts described above do not necessarily have to be processed in chronological order in the order described in the flowcharts, but are performed in parallel or individually (for example, parallel processes or objects). Processing).

  The embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention.

It is a figure for demonstrating the conventional image processing apparatus. It is a figure for demonstrating the image obtained by the conventional image processing. It is a block diagram which shows the structural example of the image processing apparatus of one Embodiment to which this invention is applied. It is a figure which shows the image obtained at each process of the conventional image processing. It is a figure explaining the correction process of the conventional correction process part. It is a block diagram which shows the example of a functional structure of the conventional reverse gamma process part and a correction process part. 4 is a flowchart illustrating image processing by the image processing apparatus in FIG. 3. It is a figure which shows the example of the characteristic of reverse gamma. It is a block diagram which shows another example of a functional structure of the conventional reverse gamma process part, and the example of a functional structure of the conventional correction process part. FIG. 7 is a block diagram illustrating an example of functional configurations of an inverse gamma processing unit and a correction processing unit to which the present invention is applied with respect to FIG. 6. It is a block diagram which shows the example of a functional structure of the gain part of the correction process part of FIG. It is a figure which shows an example of the function for the gain part of FIG. 11 to determine a gain. FIG. 7 is a block diagram illustrating an example of functional configurations of an inverse gamma processing unit and a correction processing unit to which the present invention is applied with respect to FIG. 6. And FIG. 11 is a block diagram illustrating an example of a configuration of a computer that executes a series of processes by a program.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 11 Camera, 12 Image processing apparatus, 13 Display, 14 Imaging part, 15 Gamma processing part, 21 Image processing apparatus, 22 Inverse gamma processing part, 23 Correction processing part, 24 Gamma processing part, 32A, 33A Calculation part, 33 Replacement part , 34 Gamma characteristic adjustment unit, 81 gain unit, 82 subtraction unit, 83 multiplication unit, 84 addition unit, 91 subtraction unit, 92 fixed value output unit, 93 absolute value calculation unit, 94 function unit, 101 CPU, 102 ROM, 103 RAM, 108 storage unit, 111 removable media

Claims (4)

An inverse gamma processing means for obtaining the luminance signal and performing inverse gamma processing on the luminance signal from an imaging device that performs gamma processing on the luminance signal of an image obtained as a result of imaging the subject and outputs the luminance signal;
For the luminance signal subjected to the inverse gamma processing by the inverse gamma processing means, the degree of gamma processing is adjusted based on the color information of the image, and the luminance signal after adjustment is Correction processing means for performing correction processing for correcting blur of the image ,
The correction processing means includes
Gamma characteristic adjusting means for adjusting the degree of gamma processing on the luminance signal subjected to the reverse gamma processing by the reverse gamma processing means, based on the color information of the image;
Of the pixel values constituting the luminance signal after adjustment by the inverse gamma characteristic adjusting means, the pixel value of the pixel of interest, the pixel value of a pixel spaced in one direction from the pixel of interest, and other values from the pixel of interest A calculation means for performing a predetermined calculation using pixel values of pixels separated in a direction;
Based on the calculation result by the calculation means, the pixel value of the target pixel is selected from the pixel value of the pixel spaced in one direction from the target pixel or the pixel value of the pixel spaced in the other direction from the target pixel. Replacement means to replace with
An image processing apparatus. The image processing apparatus according to claim 1, further comprising: a gamma processing unit that performs the same gamma processing as the gamma processing performed in the imaging device on the luminance signal that has been subjected to the correction processing by the correction processing unit. As steps executed by the image processing apparatus,
From the imaging device that outputs by performing gamma processing on the luminance signal of the image obtained as a result of imaging the subject, obtains the luminance signal, performs inverse gamma processing on the luminance signal,
For the luminance signal subjected to the inverse gamma processing, the degree of gamma processing is adjusted based on the color information of the image, and the blur of the image is corrected with respect to the adjusted luminance signal. It includes facilities to step the correction process,
The correction process includes
For the luminance signal that has been subjected to the inverse gamma process by the inverse gamma process, a gamma characteristic adjustment process that adjusts the degree of the gamma process based on the color information of the image is performed,
Of the pixel values constituting the luminance signal after adjustment by the inverse gamma characteristic adjustment processing, the pixel value of the target pixel, the pixel value of the pixel spaced in one direction from the target pixel, and the other from the target pixel Using the pixel values of pixels separated in the direction, perform a calculation process to perform a predetermined calculation,
Based on the calculation result of the calculation process, the pixel value of the target pixel is selected from the pixel value of the pixel spaced in one direction from the target pixel, or the pixel value of the pixel spaced in the other direction from the target pixel. An image processing method including a step of performing a replacement process to be replaced with any one . As steps that the computer performs:
From the imaging device that outputs by performing gamma processing on the luminance signal of the image obtained as a result of imaging the subject, obtains the luminance signal, performs inverse gamma processing on the luminance signal,
For the luminance signal subjected to the inverse gamma processing, the degree of gamma processing is adjusted based on the color information of the image, and the blur of the image is corrected with respect to the adjusted luminance signal. Including a step of performing a correction process ,
The correction process includes
For the luminance signal that has been subjected to the inverse gamma process by the inverse gamma process, a gamma characteristic adjustment process that adjusts the degree of the gamma process based on the color information of the image is performed,
Of the pixel values constituting the luminance signal after adjustment by the inverse gamma characteristic adjustment processing, the pixel value of the target pixel, the pixel value of the pixel spaced in one direction from the target pixel, and the other from the target pixel Using the pixel values of pixels separated in the direction, perform a calculation process to perform a predetermined calculation,
Based on the calculation result of the calculation process, the pixel value of the target pixel is selected from the pixel value of the pixel spaced in one direction from the target pixel, or the pixel value of the pixel spaced in the other direction from the target pixel. A program including a step of performing a replacement process to be replaced with one .

Images