JP4210950B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method, and more particularly to an image processing apparatus and an image processing method suitable for use when converting a standard resolution image into a high resolution image.

  For example, when converting a standard resolution or low resolution image (hereinafter referred to as an SD image as appropriate) to a high resolution image (hereinafter referred to as an HD image as appropriate) or enlarging the image, Interpolation (compensation) of the pixel values of the missing pixels is performed by a so-called interpolation filter or the like.

  However, even if pixel interpolation is performed using an interpolation filter, it is difficult to obtain a high-resolution image because the HD image component (high-frequency component) that is not included in the SD image cannot be restored.

  Therefore, the applicant of the present application has previously proposed an image conversion apparatus that converts an SD image into an HD image that also includes a high-frequency component not included therein.

  In this image conversion apparatus, high-frequency components not included in the SD image are restored by performing an adaptive process for obtaining a prediction value of a pixel of the HD image by linear combination of the SD image and a predetermined prediction coefficient. It is made so that.

That is, for example, the predicted value E [y] of the pixel value y of a pixel constituting the HD image (hereinafter referred to as HD pixel as appropriate) is set to the pixel value of several SD pixels (pixels constituting the SD image). (Hereinafter referred to as “learning data” where appropriate) It is considered to obtain by a linear first combination model defined by a linear combination of x ₁ , x ₂ ,... And predetermined prediction coefficients w ₁ , w ₂ ,. . In this case, the predicted value E [y] can be expressed by the following equation.

E [y] = w ₁ x ₁ + w ₂ x ₂ +...
... (1)

  Therefore, in order to generalize, a matrix W composed of a set of prediction coefficients w, a matrix X composed of a set of learning data, and a matrix Y ′ composed of a set of predicted values E [y],

で定義すると、次のような観測方程式が成立する。

Then, the following observation equation holds.

XW = Y '
... (2)

  Then, it is considered to apply the least square method to this observation equation to obtain a predicted value E [y] close to the pixel value y of the HD pixel. In this case, a matrix Y composed of a set of true pixel values y of HD pixels serving as teacher data and a matrix E composed of a set of residuals e of predicted values E [y] with respect to the pixel values y of HD pixels,

で定義すると、式（２）から、次のような残差方程式が成立する。

From the equation (2), the following residual equation is established.

XW = Y + E
... (3)

In this case, the prediction coefficient w _i for obtaining the prediction value E [y] close to the pixel value y of the HD pixel is, for example, a square error

を最小にすることで求めることができる。

Can be obtained by minimizing.

Therefore, in order to obtain the prediction value E [y] near the pixel value y of the HD pixel, when the above-mentioned square error differentiated by the prediction coefficient w _i is 0, that is, the prediction coefficient w _i satisfying the following equation: That is the optimum value.

・・・（４）

... (4)

Therefore, first, the following equation is established by differentiating the equation (3) by the prediction coefficient w _i .

・・・（５）

... (5)

  From equations (4) and (5), equation (6) is obtained.

・・・（６）

... (6)

  Further, considering the relationship among the learning data x, the prediction coefficient w, the teacher data y, and the residual e in the residual equation of Equation (3), the following normal equation can be obtained from Equation (6). .

・・・（７）

... (7)

  The normal equation of the equation (7) can be formed by the same number as the number of prediction coefficients w to be obtained. Therefore, by solving the equation (7) (however, to solve the equation (7), the equation (7) 7), the matrix composed of the coefficients related to the prediction coefficient w needs to be regular), and the optimal prediction coefficient w can be obtained. In solving equation (7), for example, a sweep-out method (Gauss-Jordan elimination method) or the like can be applied.

  As described above, the optimum prediction coefficient w is obtained, and further, using the prediction coefficient w, it is adaptive to obtain the prediction value E [y] close to the pixel value y of the HD pixel by the equation (1). It is processing.

  Note that the adaptive processing is different from the interpolation processing in that a component included in the HD image that is not included in the SD image is reproduced. In other words, the adaptive process is the same as the interpolation process using a so-called interpolation filter as long as only Expression (1) is seen, but the prediction coefficient w corresponding to the tap coefficient of the interpolation filter uses the teacher data y. In other words, since it is obtained by learning, the components included in the HD image can be reproduced. That is, a high-resolution image can be easily obtained. From this, it can be said that the adaptive process is a process having an effect of creating an image (its resolution).

  FIG. 12 shows a configuration example of an image conversion apparatus that converts an SD image into an HD image by the adaptive processing as described above.

  The SD image is supplied to the class classification unit 201 and the adaptive processing unit 204. The class classification unit 201 includes a class tap generation circuit 202 and a class classification circuit 203, in which an HD pixel (a focused HD pixel) for which a predicted value is to be obtained by adaptive processing (hereinafter referred to as a focused pixel as appropriate). Is classified into a predetermined class based on the property of the pixel of the SD image corresponding to the target pixel.

  That is, the class tap generation circuit 202 is used to classify the pixel of interest. As SD pixels corresponding to the target pixel (hereinafter referred to as class taps as appropriate), for example, a plurality of SD pixels having a predetermined positional relationship with respect to the target pixel are extracted from the SD image supplied to the class classification unit 201. And supplied to the class classification circuit 203. The class classification circuit 203 detects the pixel value pattern (pixel value distribution) of the SD pixels constituting the class tap from the class tap generation circuit 202, and the value assigned in advance to the pattern is used as the class of the target pixel. To the adaptive processing unit 204.

Specifically, for example, now, an HD image is composed of pixels (HD pixels) indicated by x in FIG. 13, and an SD image is composed of pixels (SD pixels) indicated by ◯ in FIG. Let's say. In FIG. 13, the SD image is configured by halving the number of horizontal or vertical pixels of the HD image. Here, in FIG. 13 (the same applies to FIGS. 14 to 16 described later), the i + 1th SD pixel from the left and the j + 1th SD pixel from the top (the portion indicated by a circle in the figure) is represented as X _{i, j.} Similarly, the i ′ + 1-th HD pixel from the left and the j ′ + 1-th HD pixel from the top (portion indicated by a cross in the figure) is represented as Y _{i ′, j ′} . In this case, the position of the SD pixel X _{i, j} coincides with the position of the HD pixel Y _{2i, 2j} .

_Assuming that an HD pixel Y _4,4 that coincides with the position of X _2,2 , for example, as a certain SD pixel is a pixel of interest, the class tap generation circuit 202 uses the SD pixel corresponding to the HD pixel Y _4,4. as, for example, SD HD pixels Y _{4, 4} and 3 × 3 centered on the SD pixels X _{2, 2} correlation coincides with the position of HD pixel Y _{4, 4} which are expected to be high for (horizontal × vertical) Pixels X _1,1 , X _2,1 , X _3,1 , X _1,2 , X _2,2 , X _3,2 , X _1,3 , X _2,3 , X _3,3 (dotted line in FIG. 13) SD pixels in a range surrounded by) are extracted, and are used as class taps of the _target pixel (HD pixel) _Y4,4 .

For example, when the HD pixel Y _5,4 right next to the HD pixel Y _4,4 that coincides with the position of X _2,2 is set as the target pixel, the class tap generation circuit 202 causes the HD pixel Y _{5,4 to} be the target pixel. _As SD pixels corresponding to ₅ and ₄ , for example, as shown by being surrounded by a dotted line in FIG. 14, SD pixels X _{1 and} X ₁ in the class tap formed when the HD pixel Y _{4 and 4} is set as the target pixel. Instead of ₂ pixels, those including SD pixels X ₄ and ₂ are extracted, and the 9 SD pixels are class taps of the _target pixel (HD pixel) Y _{5 and 4} .

Further, for example, when the HD pixel Y _4,5 adjacent below the HD pixel Y _4,4 coinciding with the position of X _2,2 is set as the target pixel, the class tap generation circuit 202 causes the HD pixel As the SD pixel corresponding to Y _4,5 , for example, as shown by being surrounded by a dotted line in FIG. 15, the SD pixel X ₂ in the class tap formed when the HD pixel Y _4,4 is set as the target pixel. _{, 1} are extracted including the SD pixels X _2,4 , and the nine SD pixels are used as class taps of the _target pixel (HD pixel) Y _4,5 .

Further, for example, when the HD pixel Y _5,5 adjacent to the lower right of the HD pixel Y _4,4 that coincides with the position of X _2,2 is the target pixel, the class tap generation circuit 202 As the SD pixels corresponding to the HD pixels Y _5,5 , for example, as shown by being surrounded by a dotted line in FIG. 16, the SD pixels in the class tap formed when the HD pixel Y _4,4 is the target pixel. Instead of X _1,1 , a pixel including SD pixels X _4,4 is extracted, and the nine SD pixels are used as class taps of the _target pixel (HD pixel) Y _5,5 .

  The class classification circuit 203 detects a pattern of nine SD pixels (pixel values) as the class tap configured by the class tap generation circuit 202, and outputs a value corresponding to the pattern as a class of the target pixel. Is done.

  This class is supplied to an address terminal (AD) of a coefficient ROM (Read Only Memory) 207 in the adaptive processing unit 204.

Here, 8 bits or the like are generally assigned to the pixels constituting the image. Assuming that 8 bits are allocated to the SD pixel, for example, even if only the 3 × 3 pixel square class tap shown in FIG. 13 is considered, the number of pixel value patterns is (2 ⁸ ) ^9. The number of streets becomes enormous, and it is difficult to speed up subsequent processing.

  Therefore, as preprocessing before class classification, the class tap is subjected to, for example, ADRC (Adaptive Dynamic Range Coding) processing, which is processing for reducing the number of bits of SD pixels constituting the class tap.

That is, in the ADRC processing, first, from the nine SD pixels constituting the class tap, those having the maximum pixel value (hereinafter referred to as the maximum pixel as appropriate) and those having the minimum value (hereinafter referred to as the minimum pixel as appropriate). Is detected. Then, a difference DR (= MAX−MIN) between the pixel value MAX of the maximum pixel and the pixel value MIN of the minimum pixel is calculated, and this DR is set as a local dynamic range of the class tap. Based on the dynamic range DR, Each pixel value constituting the class tap is requantized to K bits smaller than the original number of assigned bits. That is, the pixel value MIN of the minimum pixel from each pixel value is subtracted forming the class taps, each subtraction value is divided by DR / 2 ^K.

As a result, each pixel value constituting the class tap is expressed by K bits. Therefore, for example, when K = 1, the number of patterns of the pixel values of the nine SD pixels is (2 ¹ ) ⁹ and the number of patterns is very small compared to the case where ADRC processing is not performed. It can be.

  On the other hand, the adaptive processing unit 204 includes a prediction tap generation circuit 205, a prediction calculation circuit 206, and a coefficient ROM 207, where adaptive processing is performed.

  That is, in the prediction tap generation circuit 205, a plurality of pixels having a predetermined positional relationship with respect to the target pixel used in the prediction calculation circuit 206 to obtain the predicted value of the target pixel from the SD image supplied to the adaptive processing unit 204. SD pixels are extracted and supplied to the prediction calculation circuit 206 as prediction taps.

Specifically, for example, HD pixel Y _{4, 4} is a pixel of interest, when the class tap as described in FIG. 13 constituted, the prediction tap generating circuit 205, for example, an HD pixel Y _{4, 4} 5 × 5 SD pixels centered on the SD pixel X _2,2 that coincides with the position of the target pixel Y _{4,4 in} the range surrounded by the solid line in FIG. Are extracted and used as prediction taps for the target pixel (HD pixel) Y _4,4 .

For example, when the HD pixel Y _5,4 is set as the target pixel, the prediction tap generation circuit 205 sets the HD pixel Y _{4,4 as} the target pixel, for example, as shown by a solid line in FIG. Then, instead of the SD pixel X _0,2 in the prediction tap formed, the one including the SD pixel X _5,2 is extracted, and the 25 SD pixels are the target pixel (HD pixel) The class tap is Y _5,4 .

Further, for example, when the HD pixel Y _4,5 is set as the target pixel, the prediction tap generation circuit 205 sets the HD pixel Y _{4,4 as} the target pixel as shown by a solid line in FIG. In the prediction tap formed in this case, SD pixels X _2,5 are extracted instead of the SD pixels X _2,0 , and the 25 SD pixels are extracted as the target pixel (HD pixel). The prediction tap is Y _4,5 .

For example, when the HD pixel Y _5,5 is set as the target pixel, the prediction tap generation circuit 205 sets the HD pixel Y _{4,4 as} the target pixel as shown by a solid line in FIG. 16, for example. In the class tap formed in this case, SD pixels X _5,5 are extracted instead of the SD pixels X _0,0 , and the 25 SD pixels are extracted as _target pixels (HD pixels). The prediction tap is Y _5,5 .

  The prediction calculation circuit 206 is supplied with the prediction coefficient from the coefficient ROM 207 in addition to the prediction tap from the prediction tap generation circuit 205.

  That is, the coefficient ROM 207 stores the prediction coefficient obtained by learning in advance for each class. When a class is supplied from the class classification circuit 203, the coefficient ROM 207 is stored at an address corresponding to the class. The prediction coefficient is read out and supplied to the prediction calculation circuit 206.

As a result, the prediction calculation circuit 206 is supplied with the prediction tap corresponding to the target pixel and the prediction coefficient for the class of the target pixel. In the prediction arithmetic circuit 206, the prediction coefficients w, w ₂ ,... From the coefficient ROM 207 and the prediction taps (constituting SD pixels) x ₁ , x ₂ ,. Is used to calculate the prediction value E [y] of the pixel of interest (HD pixel) y, and this is output as the pixel value of the HD pixel.

  The above processing is performed with all the HD pixels as the target pixel, whereby the SD image is converted into an HD image. Note that the class tap generation circuit 202 and the prediction tap generation circuit 205 perform processing using the same HD pixel as the target pixel.

  Next, FIG. 17 shows a configuration example of a learning apparatus that performs a learning process for calculating a prediction coefficient to be stored in the coefficient ROM 207 of FIG.

  The HD image to be the teacher data y in learning is supplied to the thinning circuit 211 and the teacher data extraction circuit 146. In the thinning circuit 211, for example, the number of pixels of the HD image is thinned out. Thus, an SD image is obtained. That is, in the thinning circuit 211, the number of horizontal or vertical pixels of the HD image is halved, thereby forming an SD image. The SD image is supplied to the class classification unit 212 and the prediction tap generation circuit 145.

  In the class classification unit 212 or the prediction tap generation circuit 145, the same processing as that in the class classification unit 201 or the prediction tap generation circuit 205 in FIG. 12 is performed, whereby the class or prediction tap of the pixel of interest is output, respectively. . The class output from the class classification unit 212 is supplied to the prediction tap memory 147 and the address terminal (AD) of the teacher data memory 148, and the prediction tap output from the prediction tap generation circuit 145 is supplied to the prediction tap memory 147. Note that the class classification unit 212 and the prediction tap generation circuit 145 perform processing using the same HD pixel as the pixel of interest.

  The prediction tap memory 147 stores the prediction tap supplied from the prediction tap generation circuit 145 at an address corresponding to the class supplied from the class classification unit 212.

  On the other hand, in the teacher data extraction circuit 146, the HD pixel that is the target pixel in the class classification unit 212 and the prediction tap generation circuit 145 is extracted from the HD image supplied thereto, and is supplied to the teacher data memory 148 as teacher data. Is done.

  The teacher data memory 148 stores the teacher data supplied from the teacher data extraction circuit 146 at the address corresponding to the class supplied from the class classification unit 212.

  The above processing is sequentially performed on all HD pixels constituting all HD images prepared for learning in advance as the target pixel.

  As a result, at the same address in the teacher data memory 148 or the prediction tap memory 147, the HD pixel of the class corresponding to the address, or the position that constitutes the prediction tap described in FIGS. A certain SD pixel is stored as teacher data y or learning data x, respectively.

  The prediction tap memory 147 and the teacher data memory 148 can store a plurality of pieces of information at the same address, whereby a plurality of learnings classified into the same class are stored at the same address. Data x and teacher data y can be stored.

  Thereafter, the arithmetic circuit 149 reads prediction pixels as learning data or HD pixels as teacher data stored in the same address from the prediction tap memory 147 or the teacher data memory 148, and uses them, for example, the minimum self A prediction coefficient that minimizes an error between the predicted value and the teacher data is calculated by multiplication. That is, in the arithmetic circuit 149, the normal equation shown in Expression (7) is established for each class, and the prediction coefficient is obtained by solving this.

  As described above, the prediction coefficient for each class obtained by the arithmetic circuit 149 is stored in the address corresponding to each class in the coefficient ROM 207 of FIG.

  In the learning process as described above, there may occur a class in which the number of normal equations necessary for obtaining the prediction coefficient cannot be obtained. For such a class, for example, the class is ignored and the normal equation is ignored. The prediction coefficient obtained by solving the above is used as a default prediction coefficient.

  According to the adaptive processing, it is possible to obtain an HD image including a high-frequency component that is not included in the original SD image. Further, by performing class classification processing on the target pixel and performing adaptive processing using the prediction coefficient corresponding to the class obtained as a result, the adaptive processing suitable for the target pixel can be performed.

  By the way, in the above-described image conversion device (FIG. 12) and learning device (FIG. 17), the class tap is performed on the target pixel regardless of the characteristics of the image, for example, as described in FIGS. It is composed of SD pixels in a positional relationship.

  That is, the class tap is configured by nine SD pixels near the target pixel, as described with reference to FIGS.

  On the other hand, an image can be divided into areas composed of pixels with approximate luminance values, colors, and the like, that is, areas where predetermined objects are displayed, areas having predetermined colors and patterns, and the like. Therefore, when class classification as described above is performed, even in different regions in the image, for example, in a portion where there is almost no change in pixels, the target pixel is all classified into the same class. There is.

  However, even in such a portion, for example, by classifying by classifying SD pixels that are somewhat separated from the target pixel into the class tap, the target pixels that have been classified into the same class are changed to different classes. May be classified. That is, by configuring a class tap with a wider range of SD pixels, the target pixel may be classified into a class that matches the target pixel.

  If the target pixel can be classified into the class according to the class, it is possible to perform more suitable adaptive processing on the target pixel, and improve the image quality of the resulting HD image. Is possible.

  The present invention has been made in view of such circumstances, and is intended to improve image quality.

  The image processing apparatus according to the first aspect of the present invention includes a second image in which the first image is represented by a smaller number of pixels than the number of pixels of the first image, and the number of pixels of the second image. An image processing apparatus that performs a process for obtaining the first image using a third image represented by a small number of pixels, and the pixel of the first image of interest is a pixel of interest. Activity detecting means for detecting an activity of a portion corresponding to the target pixel in the second and third images, and an image having an activity exceeding a predetermined threshold among the second and third images. A selection means for selecting, and a value corresponding to a pixel value pattern of a pixel in a predetermined area centered on a pixel of the selected image corresponding to the target pixel, as a class, and Class to classify Class classification means for performing classification, and prediction means for predicting the predicted value of the pixel of interest corresponding to the class of the pixel of interest, wherein the prediction means includes the pixels of the second and third images. Prediction coefficient storage means for storing a prediction coefficient for calculating a prediction value of the target pixel by linear combination for each class, the prediction coefficient for the class of the target pixel, the second and third Prediction value calculation means for obtaining a predicted value of the target pixel by linearly combining pixel values of a plurality of pixels having a predetermined positional relationship with respect to the target pixel. Features.

  The value corresponding to the pixel value pattern of the pixels in the predetermined area of the selected image can be obtained by ADRC processing the pixel values of the pixels in the predetermined area.

  The second and third images have a smaller number of pixels than the first image by adding pixel values of adjacent pixels of the first image to a pixel value of one pixel. It can be an image.

  The image processing method according to the first aspect of the present invention is based on a second image in which the first image is represented by a smaller number of pixels than the number of pixels of the first image, and the number of pixels of the second image. And a third image represented by a small number of pixels, and an image processing method for performing processing for obtaining the first image, wherein the pixel of the first image of interest is a pixel of interest Detecting an activity of a portion corresponding to the target pixel in the second and third images, selecting an image having an activity exceeding a predetermined threshold from the second and third images, and By classifying a value corresponding to a pixel value pattern of a pixel in a predetermined area centered on a pixel of the selected image corresponding to the target pixel, the target pixel is classified into a predetermined class, and the target The predicted value of the pixel, the class of the pixel of interest When predicting correspondingly and predicting the predicted value of the target pixel, a prediction coefficient for calculating the predicted value of the target pixel is calculated for each class by linear combination with the pixels of the second and third images. A plurality of the prediction coefficients for the class of the target pixel acquired from the stored prediction coefficient storage means and a plurality of pixels having a predetermined positional relationship with respect to the target pixel among the pixels of the second and third images The prediction value of the target pixel is obtained by linearly combining the pixel value of the pixel of interest.

  In the first aspect of the present invention, the second image represented by the number of pixels smaller than the number of pixels of the first image, wherein the pixel of the first image of interest is the pixel of interest, and the second Activity of a portion corresponding to the target pixel in the third image represented by the number of pixels smaller than the number of pixels of the image of the first image, and an activity exceeding a predetermined threshold among the second and third images An image having is selected. Further, by classifying a value corresponding to a pixel value pattern of a pixel in a predetermined area centered on a pixel of the selected image corresponding to the target pixel, the target pixel is classified into a predetermined class. The When predicting the predicted value of the target pixel corresponding to the class of the target pixel, a prediction coefficient for calculating the predicted value of the target pixel by linear combination with the pixels of the second and third images The prediction coefficient for the class of the target pixel acquired from the prediction coefficient storage means stored for each class, and a predetermined position with respect to the target pixel among the pixels of the second and third images Pixel values of a plurality of related pixels are linearly combined, and a predicted value of the target pixel is obtained.

  The image processing apparatus according to the second aspect of the present invention includes a second image in which the first image is represented by a smaller number of pixels than the number of pixels of the first image, and the number of pixels of the second image. Using the third image represented by a smaller number of pixels and the fourth image represented by the number of pixels smaller than the number of pixels of the third image, a process for obtaining the first image is performed. An activity detection means for detecting an activity of a portion corresponding to the target pixel in the second to fourth images, wherein the pixel of the first image of interest is the target pixel; Selection means for selecting a plurality of images having an activity exceeding a predetermined threshold from the second to fourth images, and a predetermined centering on pixels of the selected plurality of images corresponding to the target pixel Corresponds to the pixel value pattern of the pixels in the region And classifying means for classifying the pixel of interest into a predetermined class represented by the bit sequence by making each bit string, and the predicted value of the pixel of interest Prediction means for predicting corresponding to a class, wherein the prediction means includes a prediction coefficient for calculating a prediction value of the pixel of interest by linear combination with pixels of the second to fourth images. A plurality of prediction coefficient storage means stored for each, the prediction coefficient for the class of the target pixel, and a plurality of pixels of the second to fourth images that are in a predetermined positional relationship with the target pixel Prediction value calculation means for obtaining a prediction value of the target pixel by linearly combining the pixel value of the pixel of interest.

  The value corresponding to the pixel value pattern of the pixels in the predetermined region of the selected plurality of images can be obtained by performing ADRC processing on the pixel values of the pixels in the predetermined region.

  The second to fourth images have a smaller number of pixels than the first image by adding pixel values of adjacent pixels of the first image to a pixel value of one pixel. It can be an image.

  In the image processing method according to the second aspect of the present invention, the first image is represented by a second image in which the number of pixels is smaller than the number of pixels of the first image, and the number of pixels of the second image. Using the third image represented by a smaller number of pixels and the fourth image represented by the number of pixels smaller than the number of pixels of the third image, a process for obtaining the first image is performed. In the image processing method, the pixel of the first image being noticed is set as the pixel of interest, the activity of the portion corresponding to the pixel of interest in the second to fourth images is detected, and the second A pixel value of a pixel in a predetermined area centering on a pixel of the selected plurality of images corresponding to the target pixel is selected from among the fourth to fourth images having an activity exceeding a predetermined threshold. Each value corresponding to the pattern is acquired and converted into one bit string. By classifying the target pixel into a predetermined class represented by the bit string, predicting the predicted value of the target pixel corresponding to the class of the target pixel, and predicting the predicted value of the target pixel The pixel of interest acquired from the prediction coefficient storage means for storing the prediction coefficient for calculating the prediction value of the pixel of interest by linear combination with the pixels of the second to fourth images for each class. The target pixel is obtained by linearly combining the prediction coefficient for the class and pixel values of a plurality of pixels having a predetermined positional relationship with respect to the target pixel among the pixels of the second to fourth images. It is characterized in that a predicted value of is obtained.

  In the second aspect of the present invention, the second image represented by the number of pixels smaller than the number of pixels of the first image, where the pixel of the first image of interest is the pixel of interest, Activity of a portion corresponding to the target pixel in the third image represented by the number of pixels smaller than the number of pixels of the image and the fourth image represented by the number of pixels smaller than the number of pixels of the third image Is detected, and a plurality of images having activities exceeding a predetermined threshold are selected from the second to fourth images. Then, each of the values corresponding to the pixel value pattern of the pixels in the predetermined area centered on the selected pixels of the plurality of images corresponding to the target pixel is obtained and converted into one bit string. Pixels are classified into a predetermined class represented by the bit string. When predicting the predicted value of the target pixel corresponding to the class of the target pixel, a prediction coefficient for calculating the predicted value of the target pixel by linear combination with the pixels of the second to fourth images The prediction coefficient for the class of the target pixel acquired from the prediction coefficient storage means stored for each class, and a predetermined position with respect to the target pixel among the pixels of the second to fourth images Pixel values of a plurality of related pixels are linearly combined, and a predicted value of the target pixel is obtained.

  According to the present invention, the target pixel that is the pixel of the first image of interest can be classified into a more suitable class, and the image quality of the HD image can be improved.

  FIG. 1 shows a configuration example of an embodiment of an image processing apparatus to which the present invention is applied.

  In this image processing apparatus, by processing a plurality of SD images (second and third images) having different numbers of pixels, the number of pixels is larger than the SD image (here, the resolution is also higher). An HD image (first image) is generated.

  That is, the first to third hierarchy SD images are stored in the first hierarchy memory 101, the second hierarchy memory 102, and the third hierarchy memory 103. When the SD image of the first hierarchy is used as a reference, the SD image of the second hierarchy is obtained by reducing the number of pixels of the SD image of the first hierarchy, and the SD image of the third hierarchy is the second hierarchy. The number of pixels of the SD image is reduced.

  SD image of the first hierarchy stored in the first hierarchy memory 101, SD image of the second hierarchy stored in the second hierarchy memory 102, and SD of the third hierarchy stored in the third hierarchy memory 103 All the images are supplied to the multi-layer class classification unit 104. Further, the SD image of the first hierarchy stored in the first hierarchy memory 101 is also supplied to the adaptive processing unit 105.

  The multi-hierarchy class classifying unit 104 includes multi-hierarchy class classifying circuits 104a to 104d, and the target pixel, which is an HD pixel for which a prediction value is to be obtained by adaptive processing, is the first hierarchic memory 101, the second hierarchic memory 102, Of the pixels constituting the first to third layer images from the third layer memory 103, the pixel corresponding to the target pixel is used for multi-layer class classification, which is one of the class classifications.

  Here, in the present embodiment, for example, an HD image composed of the number of pixels obtained by doubling the number of horizontal or vertical pixels of the first layer SD image is generated. . In this case, as described with reference to FIGS. 13 to 16, for an SD pixel (here, the SD pixel in the first layer), an HD pixel arranged at the same position as the SD pixel, It is necessary to generate a total of four HD pixels including three HD pixels adjacent to the right, lower, and diagonally lower right of the HD pixel. That is, it is necessary to generate four HD pixels for one SD pixel in the first layer. Therefore, in order to simultaneously perform the four HD pixel class classification processing (here, the multi-layer class classification processing), the multi-layer class classification unit 104 includes four multi-layer class classification circuits 104a to 104d. Yes.

  All of the class classification results of the four target pixels in the multi-layer class classification circuits 104 a to 104 d are supplied to the adaptive processing unit 105. The adaptive processing unit 105 is also composed of four adaptive processing circuits 105a to 105d for the same reason that the multi-layer class classification unit 104 is composed of four multi-layer class classification circuits 104a to 104d. In each of the four adaptive processing circuits 105a to 105d, adaptive processing is performed corresponding to each of the class classification results from the multi-layer class classification circuits 104a to 104d, and the predicted values of the four target pixels are obtained. The prediction values obtained by the adaptive processing circuits 105a to 105d are supplied to and stored in the HD image memory 106.

  That is, the HD image memory 106 includes four memories 106 a to 106 d as in the case of the multi-layer class classification unit 104 and the adaptive processing unit 105. Then, the prediction values supplied from the adaptive processing units 105a to 105d are stored in the memories 106a to 106d, respectively.

  Here, with respect to an SD pixel in the first hierarchy, for an HD pixel arranged at the same position as the SD pixel, for example, in the multi-hierarchy class classification circuit 104a or the adaptive processing circuit 105a, An adaptive process is performed, and the predicted value is stored in the memory 106a. In addition, with respect to an SD pixel in the first hierarchy, an HD pixel adjacent to the right, lower, or diagonally lower right of the HD pixel arranged at the same position as the SD pixel, the multi-layer class classification circuit 104b and Processing is performed in the adaptive processing circuit 105b, the multi-layer class classification circuit 104c and the adaptive processing circuit 105c, or the multi-layer class classification circuit 104d and the adaptive processing circuit 105d, and the respective predicted values obtained as a result are stored in the memories 106b to 106d. To be remembered.

  Next, the details of the multi-level class classifying unit 104 and the adaptive processing unit 15 will be described. Before that, the first level memory 101, the second level memory 102, and the third level memory 103 are stored in the first level memory A method for generating the SD image of the third layer will be described.

  FIG. 2 shows a configuration example of a storage device that generates SD images of the first to third layers.

  This storage device is composed of, for example, a one-chip CMOS (Complementary Metal Oxide Semiconductor), and the SD image input thereto is used as a first-level SD image and has a number of pixels higher than that of the first-level SD image. Three-level hierarchical encoding is performed to generate a small second SD image and a third SD image having a smaller number of pixels than the second SD image.

  That is, the address supply circuit 1 is supplied with a horizontal address or a vertical address as an address corresponding to the position in the horizontal direction or the vertical direction of the SD pixel constituting the image input to the storage device. .

In this embodiment, for example, as shown in FIG. 3, an image (digital image data) in which one screen is composed of 512 pixels in the horizontal direction and 512 lines in the vertical direction is an SD image in the first layer. As input. Therefore, both the horizontal address and the vertical address are represented by 9 (= log ₂ 512) bits.

  In the present embodiment, as described above, an HD image in which the number of horizontal or vertical pixels of the SD image in the first layer is doubled is generated. Therefore, one screen of the HD image is 1024 × It is composed of 1024 pixels.

  The address supply circuit 1 processes the horizontal address and the vertical address supplied to the address supply circuit 1 as necessary, and supplies the processed addresses to the first hierarchy memory 2, the second hierarchy memory 3, and the third hierarchy memory 4. Yes. In addition to the horizontal address and the vertical address, the address supply circuit 1 is also supplied with a clock (not shown in FIGS. 4 to 6 to be described later), an R / W (Read / Write) signal, and a hierarchy flag. The address supply circuit 1 supplies addresses to the first hierarchy memory 2, the second hierarchy memory 3, and the third hierarchy memory 4 in synchronization with the clock. Further, the address supply circuit 1 processes a horizontal address and a vertical address supplied thereto corresponding to the R / W signal and the hierarchy flag. Further, the address supply circuit 1 supplies a predetermined control signal to an RMW (Read Modify Write) circuit 5 as necessary.

  Here, the R / W signal is a signal for instructing reading of image data from the storage device or writing of image data to the storage device, and the hierarchy flag is used when reading an image stored in the storage device. This is, for example, a 2-bit flag for instructing which one of the first to third layer SD images is to be read. Note that the image data is written to the first hierarchy memory 2, the second hierarchy memory 3, and the third hierarchy memory 4, for example, so that the R / W signal represents the writing. The hierarchy flag is ignored (meaningless). Further, reading is performed individually for each of the first hierarchy memory 2, the second hierarchy memory 3, and the third hierarchy memory 4, and therefore the hierarchy flag is valid only at the time of reading. However, it is possible to simultaneously read from the first hierarchy memory 2, the second hierarchy memory 3, and the third hierarchy memory 4. In this case, the hierarchy flag need not be used.

  The first hierarchical memory 2 stores the image data supplied from the RMW circuit 5 at the address specified by the address supply circuit 1, and reads out the image data stored at the address and outputs it to the RMW circuit 5. It is made to do. The first hierarchy memory 2 stores the first hierarchy SD image, that is, the image data input to the storage device here. Further, the first hierarchical memory 2 can store at least one screen of the first hierarchical SD image, that is, here, image data of 512 × 512 pixels as shown in FIG. Has been made. Further, the memory cells constituting the first hierarchy memory 2 have at least the same data length as the number of bits assigned to the pixels constituting the first hierarchy SD image. That is, when the pixels constituting the first layer SD image are represented by, for example, 8 bits, the memory cells constituting the first layer memory 2 have a data length of at least 8 bits.

The second hierarchical memory 3 stores the image data supplied from the RMW circuit 5 at the address specified by the address supply circuit 1, and reads out the image data stored at the address and outputs it to the RMW circuit 5. It is made to do. The second hierarchical memory 3 is adapted to store the second hierarchical SD image. That is, in the present embodiment, for example, an added value of four adjacent 2 × 2 (horizontal × vertical) pixels constituting the first layer SD image is set as one pixel of the second layer. The second layer memory 3 is configured to store a second layer SD image composed of such pixels. Further, the second hierarchy memory 3 has a storage capacity capable of storing at least one screen of the second hierarchy SD image. That is, here, since 1 pixel of the second layer is formed from 2 × 2 pixels of the first layer, the SD image of the second layer is 256 × 256 (= 512/2 × 512/2) pixels. Will be composed. Therefore, the second hierarchy memory 2 can store at least a second hierarchy SD image having such a number of pixels. Further, the memory cells constituting the second hierarchy memory 3 have a data length that can store at least the pixels constituting the SD image of the second hierarchy without dropping. That is, in the present embodiment, since the pixels of the first layer are represented by 8 bits, the pixels of the second layer which are four addition values of such 8-bit pixels are 10 (= log ₂ (2 ⁸ +2 ⁸ +2 ⁸ +2 ⁸ )) bits. Therefore, the memory cells constituting the second hierarchy memory 3 have a data length of at least 10 bits.

The third hierarchical memory 4 stores the image data supplied from the RMW circuit 5 at the address specified by the address supply circuit 1, and reads out the image data stored at the address and outputs it to the RMW circuit 5. It is made to do. The third hierarchical memory 4 is adapted to store the third hierarchical SD image. That is, in the present embodiment, for example, the adjacent 2 × 2 4 pixels constituting the second layer SD image, and therefore the added value of the 4 × 4 pixels constituting the first layer SD image is The third hierarchy memory 4 is configured to store a third hierarchy SD image composed of such pixels. Further, the third hierarchy memory 4 has a storage capacity capable of storing at least a third hierarchy SD image for one screen. That is, in this case, since 1 pixel of the third layer is formed from 2 × 2 pixels of the second layer, the SD image of the second layer is 128 × 128 (= 256/2 × 256/2) pixels. Will be composed. Therefore, the third hierarchy memory 4 can store at least a third hierarchy SD image composed of such a number of pixels. Further, the memory cells constituting the third hierarchy memory 4 have a data length that can store at least the pixels constituting the SD image of the third hierarchy without dropping. That is, in the present embodiment, since the second layer pixel is represented by 10 bits as described above, the third layer pixel, which is four addition values of such 10 bit pixels, is 12 (= log ₂ (2 ¹⁰ +2 ¹⁰ +2 ¹⁰ +2 ¹⁰ )) bits. Accordingly, the memory cells constituting the third hierarchy memory 4 have a data length of at least 12 bits.

  Note that a clock is supplied to the first hierarchy memory 2, the second hierarchy memory 3, and the third hierarchy memory 4, and data is read and written in synchronization with this clock. ing.

  The RMW circuit 5 writes image data supplied to the storage device in the first hierarchy memory 2 as a first hierarchy SD image. In addition, the RMW circuit 5 performs a process of calculating the second hierarchy SD image from the first hierarchy SD image and writes the second hierarchy SD image in the second hierarchy memory 3. Further, the RMW circuit 5 performs processing for calculating a third layer SD image from the first layer SD image (or second layer SD image) and writes the third layer SD image in the third layer memory 4. The RMW circuit 5 is also configured to read out and output the image data stored in the first hierarchical memory 2, the second hierarchical memory 3, or the third hierarchical memory 4, respectively. The RMW circuit 5 is supplied with a clock, an R / W signal, a hierarchy flag, and a control signal output from the address supply circuit 1. The RMW circuit 5 synchronizes with the clock, and the R / W Various processes are performed based on the signal, the hierarchy flag, and the control signal.

  Next, the operation will be described with reference to FIGS.

  Here, as shown in FIG. 3 described above, one screen is composed of 512 × 512 pixels, and image data in which each pixel is represented by 8 bits is supplied to the storage device as an SD image of the first layer. Shall be. The image data is supplied by being sequentially scanned.

Further, the pixel constituting the SD image of the first hierarchy is h (0, 0) in the upper left pixel, and the pixel in the x + 1th position from the left and the y + 1th position from the top is similarly defined as h ( x, y). Since the SD image of the first hierarchy is composed of 512 × 512 pixels as described above, x and y both take integer values in the range of 0 to 511 (= 2 ⁹ −1).

Further, considering variables s and t that take integer values in the range of 0 to 255 (= 2 ⁹ / 2-1), the pixels constituting the second hierarchy SD image are adjacent 2 × 2 pixels of the first hierarchy. An added value of the pixels h (2s, 2t), h (2s + 1, 2t), h (2s, 2t + 1), and h (2s + 1, 2t + 1) is represented by m (s, t). Therefore, the formula
m (s, t) = h (2s, 2t) + h (2s + 1,2t) + h (2s, 2t + 1) + h (2s + 1,2t + 1)
... (8)
Holds.

Further, considering the variables m and n taking integer values in the range of 0 to 127 (= 2 ⁹ / 4-1), the pixels constituting the SD image of the third hierarchy are adjacent 2 × 2 of the second hierarchy. Addition value of pixels m (2m, 2n), m (2m + 1, 2n), m (2m, 2n + 1), m (2m + 1, 2n + 1), that is, adjacent 4 × 4 pixels h (4m, 4n) in the first layer , h (4m + 1,4n), h (4m + 2,4n), h (4m + 3,4n), h (4m, 4n + 1), h (4m + 1,4n + 1), h ( 4m + 2,4n + 1), h (4m + 3,4n + 1), h (4m, 4n + 2), h (4m + 1,4n + 2), h (4m + 2,4n + 2) , h (4m + 3,4n + 2), h (4m, 4n + 3), h (4m + 1,4n + 3), h (4m + 2,4n + 3), h (4m + 3,4n The added value of +3) is expressed as q (m, n). Therefore, the formula
q (m, n) = m (2m, 2n) + m (2m + 1,2n) + m (2m, 2n + 1) + m (2m + 1,2n + 1)
= h (4m, 4n) + h (4m + 1,4n) + h (4m + 2,4n) + h (4m + 3,4n)
+ h (4m, 4n + 1) + h (4m + 1,4n + 1) + h (4m + 2,4n + 1) + h (4m + 3,4n + 1)
+ h (4m, 4n + 2) + h (4m + 1,4n + 2) + h (4m + 2,4n + 2) + h (4m + 3,4n + 2)
+ h (4m, 4n + 3) + h (4m + 1,4n + 3) + h (4m + 2,4n + 3) + h (4m + 3,4n + 3)
... (9)
Holds.

Further, the address supply circuit 1 has a combination (HA, VA) of the horizontal address HA and the vertical address VA in both cases of data writing and data reading.
(0,0), (1,0), ..., (511,0),
(0, 1), (1, 1), ..., (511, 1),
・
・
・
(511,0), (511,1), ..., (511,511)
In this order (the order corresponding to the sequential scanning), the signals are supplied in synchronization with the clock.

  Further, each bit of the 9-bit horizontal address HA is represented as ha1, ha2,..., Ha8 (ha8 is the most significant bit) with its least significant bit ha0, and each bit of the 9-bit vertical address VA. Similarly, va0 is represented as va1, va2,..., Va8 (va8 is the most significant bit) with the least significant bit as va0.

  At the time of writing image data to the storage device, the SD image of the first layer is supplied to the RMW circuit 5 while being sequentially scanned in synchronization with the clock. Assume that the address HA and the vertical address VA are supplied as described above.

  In this case, access to the first hierarchy memory 2 is performed as follows.

That is, as shown in FIG. 4, at the time of writing (when the R / W signal indicates writing), the address supply circuit 1 uses the horizontal address HA and the vertical address VA supplied thereto as they are. This is supplied to the address terminals (ADh, ADv) of the first hierarchy memory 2. On the other hand, the RMW circuit 5 converts the first layer SD image data (SD pixel (pixel value)) supplied thereto into memory cells (first layer memory 2 specified by the horizontal address HA and the vertical address VA). (Not shown). Thereafter, the same processing is performed, and the first layer SD image for one screen composed of 512 × 512 pixels is stored in the first layer memory 2. That is, as a result, the addresses (0, 0), (1, 0),..., (511, 0),
(0, 1), (1, 1), ..., (511, 1),
...
(511,0), (511,1), ..., (511,511)
Includes the first layer pixel (pixel value)
h (0,0), h (1,0),..., h (511,0),
h (0, 1), h (1, 1), ..., h (511, 1),
...
h (511,0), h (511,1),..., h (511,511)
Are stored respectively.

  At the time of reading (when the R / W signal indicates reading), the address supply circuit 1 also supplies the horizontal address HA and the vertical address VA supplied thereto if the hierarchy flag indicates the first hierarchy. Is supplied to the address terminal of the first hierarchical memory 2 as it is. Then, the RMW circuit 5 reads the first layer SD image data stored in the memory cell of the first layer memory 2 specified by the horizontal address HA and the vertical address VA, and the same processing is performed thereafter. As a result, an SD image of the first hierarchy for one screen composed of 512 × 512 pixels is read from the first hierarchy memory 2. In other words, the SD image of the first hierarchy scanned sequentially is thereby output.

  Next, access to the second hierarchy memory 3 will be described.

  First, at the time of writing, as shown in FIG. 5, for example, the address supply circuit 1 includes upper 8 bits ha1 to ha8 excluding the least significant bit ha0 as a part of the horizontal address HA supplied thereto. The upper 8 bits va1 to va8 excluding the least significant bit va0 as a part of the vertical address VA are supplied to the address terminals of the second hierarchical memory 3. Further, the address supply circuit 1 outputs the least significant bit ha0 of the horizontal address HA and the least significant bit va0 of the vertical address VA to the RMW circuit 5 as control signals.

  Therefore, for example, 2 × 2 4 pixels h (2s, 2t), h (2s + 1, 2t), h (2s, 2t + 1), h (2s + 1, 2t + 1) in the first layer as indicated by D1 in FIG. At any timing supplied to the RMW circuit 5, the address supply circuit 1 outputs a signal designating the same address (s, t) of the second hierarchy memory 3 to the second hierarchy memory 3. .

  On the other hand, in the RMW circuit 5, the first layer SD image data supplied thereto is input to the calculator 13. The computing unit 13 is supplied with the output of the switch 12 in addition to the SD image data of the first layer, and the computing unit 13 adds them and supplies them to the writing unit 14. ing.

  The switch 12 selects one of the terminals 12a and 12b corresponding to the output of the NOR gate 15, and the terminal 12a or 12b receives the output of the reading unit 11 or 0. Each is supplied. The NOR gate 15 is supplied with the least significant bit ha0 of the horizontal address HA from the address supply circuit 1 and the least significant bit va0 of the vertical address VA. Therefore, the output of the NOR gate 15 is the least significant bit. When both ha0 and va0 are 0, that is, out of 2 × 2 pixels h (2s, 2t), h (2s + 1, 2t), h (2s, 2t + 1), h (2s + 1, 2t + 1) in the first layer The pixel h (2s, 2t) at the upper left is set to the H level only at the timing when it is supplied to the calculator 13, and is set to the L level in other cases.

  The switch 12 selects the terminal 12a or 12b when the output of the NOR gate 15 is L level or H level.

  The reading unit 11 reads data (stored data) stored at an address corresponding to a signal output from the address supply circuit 1.

  Accordingly, the upper left pixel h (2s, 2t) among the 2 × 2 pixels h (2s, 2t), h (2s + 1, 2t), h (2s, 2t + 1), h (2s + 1, 2t + 1) in the first layer is obtained. At the timing supplied to the arithmetic unit 13, the data stored in the address (s, t) of the second hierarchical memory 3 is read by the reading unit 11 and supplied to the terminal 12 a. Since the least significant bit ha0 of the address HA and the least significant bit va0 of the vertical address VA are both 0, the output of the NOR gate 15 becomes H level, and the switch 12 selects the terminal 12b.

  As a result, 0 is supplied to the calculator 13 via the switch 12.

  In the arithmetic unit 13, the 0 and the first-layer pixel h (2s, 2t) are added, and the added value (0 + h (2s, 2t)) is supplied to the writing unit 14. The writing unit 14 writes the output of the arithmetic unit 13 to the address corresponding to the signal output from the address supply circuit 1, that is, the address (s, t) of the second hierarchy memory 3.

  Next, the upper left pixel h (2s, 2t) of the 2 × 2 pixels h (2s, 2t), h (2s + 1, 2t), h (2s, 2t + 1), h (2s + 1, 2t + 1) in the first layer At the timing at which the pixel h (2s + 1, 2t) on the right side of the signal is supplied to the calculator 13, the data stored in the address (s, t) of the second hierarchical memory 3 in the reading unit 11 (here, , 0 + h (2s, 2t)) is read and supplied to the terminal 12a.

  On the other hand, in this case, since the least significant bit ha0 of the horizontal address HA is 1 and the least significant bit va0 of the vertical address VA is 0, the output of the NOR gate 15 becomes L level, and the switch 12 selects the terminal 12a. To do.

  As a result, the data (stored data) read in the reading unit 11 (here, 0 + h (2s, 2t)) is supplied to the arithmetic unit 13 via the switch 12.

  In the arithmetic unit 13, the data supplied via the switch 12 is added to the pixel h (2s + 1, 2t) in the first layer, and the added value (0 + h (2s, 2t) + h (2s + 1, 2t)) is obtained. , Supplied to the writing unit 14. The writing unit 14 writes the output of the arithmetic unit 13 to the address corresponding to the signal output from the address supply circuit 1, that is, the address (s, t) of the second hierarchy memory 3.

  Thereafter, the supply of image data on the 2t + 1 line from the top of the first layer is started, and 2 × 2 pixels h (2s, 2t), h (2s + 1, 2t), h (2s, 2t + 1), h of the first layer are started. When the lower left pixel h (2s, 2t + 1) of (2s + 1, 2t + 1) is supplied to the calculator 13, it is also stored in the address (s, t) of the second hierarchical memory 3 in the reading unit 11. The data (here, 0 + h (2s, 2t) + h (2s + 1, 2t)) is read and supplied to the terminal 12a.

  On the other hand, in this case, since the least significant bit ha0 of the horizontal address HA is 0 and the least significant bit va0 of the vertical address VA is 1, the output of the NOR gate 15 becomes L level, and the switch 12 selects the terminal 12a. To do.

  As a result, the data (stored data) read in the reading unit 11 (here, 0 + h (2s, 2t) + h (2s + 1, 2t)) is supplied to the arithmetic unit 13 via the switch 12.

  The arithmetic unit 13 adds the data supplied via the switch 12 and the pixel h (2s, 2t + 1) in the first layer and adds the value (0 + h (2s, 2t) + h (2s + 1, 2t) + h ( 2s, 2t + 1)) is supplied to the writing unit 14. The writing unit 14 writes the output of the arithmetic unit 13 to the address corresponding to the signal output from the address supply circuit 1, that is, the address (s, t) of the second hierarchy memory 3.

  Next, the lower left pixel h (2s, 2t + 1) of 2 × 2 pixels h (2s, 2t), h (2s + 1, 2t), h (2s, 2t + 1), h (2s + 1, 2t + 1) in the first layer When the pixel h (2s + 1, 2t + 1) on the right side is supplied to the computing unit 13, the data stored in the address (s, t) of the second hierarchy memory 3 (here, 0 + h (2s, 2t) + h (2s + 1, 2t) + h (2s, 2t + 1)) is read and supplied to the terminal 12a.

  On the other hand, in this case, since the least significant bit ha0 of the horizontal address HA and the least significant bit va0 of the vertical address VA are both 1, the output of the NOR gate 15 becomes L level, and the switch 12 selects the terminal 12a. To do.

  As a result, the arithmetic unit 13 receives the data (stored data) read by the reading unit 11 via the switch 12 (here, 0 + h (2s, 2t) + h (2s + 1, 2t) + h (2s, 2t + 1)). ) Is supplied.

  The computing unit 13 adds the data supplied via the switch 12 and the pixel h (2s + 1, 2t + 1) in the first layer and adds the value (0 + h (2s, 2t) + h (2s + 1, 2t) + h ( 2s, 2t + 1) + h (2s + 1, 2t + 1)) is supplied to the writing unit 14. The writing unit 14 writes the output of the arithmetic unit 13 to the address corresponding to the signal output from the address supply circuit 1, that is, the address (s, t) of the second hierarchy memory 3.

  Therefore, the address (s, t) of the second hierarchical memory 3 is finally expressed by the equation h (2s, 2t) + h (2s + 1, 2t) + h (2s, 2t + 1) + h (2s + 1, 2t + 1). That is, the second layer pixel (pixel value) m (s, t) shown in the above equation (8) is stored.

  As described above, the second hierarchical memory 3 stores the second hierarchical SD image in which one screen is composed of 256 × 256 pixels.

  As described above, the SD image data of the first hierarchy is written to the address (HA, VA) of the first hierarchy memory 2 specified by the horizontal address HA and the vertical address VA, and one of the horizontal address HA and the vertical address VA. Reads out the stored data stored in the address of the second hierarchical memory specified by the sections ha1 to ha8 and va1 to va8, and adds the stored data and the first hierarchical SD image data, Since the addition value is written to the address of the second hierarchical memory where the stored data was stored, the SD image data of the second hierarchy is generated and stored at the same time as the SD image data of the first hierarchy is stored. can do. That is, SD image data of the second hierarchy can be obtained in real time.

  Next, reading of the second layer SD image from the second layer memory 3 will be described.

  At the time of reading, when the hierarchy flag indicates the second hierarchy, the address supply circuit 1 again uses the upper 8 bits ha1 to ha8 or va1 to va8 of the horizontal address HA or the vertical address VA supplied thereto, respectively. Are supplied to the address terminals of the second hierarchical memory 2, and the least significant bits ha0 or va0 are output to the RMW circuit 5 as control signals.

  On the other hand, in the RMW circuit 5, the hierarchy flag, the R / W signal, and the output of the NOR gate 15 are supplied to the reading unit 11, and the reading unit 11 reads the R / W signal. When the hierarchy flag represents the second hierarchy, the second stored in the address corresponding to the signal output from the address supply circuit 1 only when the output of the NOR gate 15 is at the H level. The SD image data of the hierarchy is read and output.

  That is, from the above, when the set of the horizontal address HA and the vertical address VA is (2s, 2t), (2s + 1, 2t), (2s, 2t + 1), (2s + 1, 2t + 1), the address supply circuit 1 All output the same address (s, t). Therefore, when data is simply read from the address of the second hierarchical memory 3 corresponding to the signal output from the address supply circuit 1, the same data is read four times.

  Therefore, in the reading unit 11, a set of the horizontal address HA and the vertical address VA is (2s, 2t), (2s + 1, 2t), (2s, 2t + 1), (2s + 1, 2t + 1), for example, (2s, 2t), that is, only when the output of the NOR gate 15 is at the H level, from the address (s, t) of the second hierarchy memory 3, the pixel (pixel value) m (s, t) of the second hierarchy Is read out.

  The second layer SD image data read by the reading unit 11 is supplied to the switch 16. The switch 16 is turned on only when the R / W signal indicates reading, and is turned off in other cases. Therefore, in this case, the switch 16 is turned on. The read second-layer SD image data is output via the switch 16.

  As described above, the second-layer SD image of one screen composed of 256 × 256 pixels stored therein is read from the second-layer memory 3. That is, the SD image of the second hierarchy scanned sequentially is thereby output.

  Next, access to the third hierarchy memory 4 will be described.

  First, at the time of writing, as shown in FIG. 6, for example, the address supply circuit 1 uses the upper 7 bits ha2 to ha2 and excluding the lower 2 bits ha0 and ha1 as part of the horizontal address HA supplied thereto. ha8 and the upper 7 bits va2 to va8 excluding the lower 2 bits va0 and va1 as a part of the vertical address VA are supplied to the address terminals of the third hierarchy memory 4. Furthermore, the address supply circuit 1 outputs the lower 2 bits ha0 and ha1 of the horizontal address HA and the lower 2 bits va0 and va1 of the vertical address VA to the RMW circuit 5 as control signals.

  Therefore, for example, as shown by D2 in FIG. 3, 4 × 4 16 pixels h (4m, 4n), h (4m + 1, 4n), h (4m + 2, 4n), h (4m + 3, 4n), h (4m, 4n + 1), h (4m + 1, 4n + 1), h (4m + 2, 4n + 1), h (4m + 3, 4n + 1), h (4m, 4n + 2), h (4m + 1, 4n + 2), h (4m + 2, 4n + 2), h (4m + 3, 4n + 2), h (4m, 4n + 3), h (4m + 1, 4n + 3), h (4m + 2, 4n + 3), h (4m + 3, 4n + 3) are supplied to the RMW circuit 5 at any timing. The address supply circuit 1 outputs a signal designating the same address (m, n) of the third hierarchy memory 4.

  On the other hand, in the RMW circuit 5, the first layer SD image data supplied thereto is input to the computing unit 23. The computing unit 23 is supplied with the output of the switch 22 in addition to the SD image data of the first layer, and the computing unit 23 adds them and supplies them to the writing unit 24. ing.

  The switch 22 selects one of the terminals 22a and 22b corresponding to the output of the NOR gate 25. The terminal 22a or 22b receives the output of the reading unit 21 or 0. Each is supplied. The NOR gate 25 is supplied with the lower 2 bits ha0 and ha1 of the horizontal address HA from the address supply circuit 1 and the lower 2 bits va0 and va1 of the vertical address VA. When the lower 2 bits ha0 and ha1 and va0 and va1 are both 0, that is, 4 × 4 pixels h (4m, 4n), h (4m + 1, 4n), h (4m + 2, 4n), h in the first layer (4m + 3, 4n), h (4m, 4n + 1), h (4m + 1, 4n + 1), h (4m + 2, 4n + 1), h (4m + 3, 4n + 1), h (4m, 4n + 2), h (4m + 1, 4n + 2), h ( 4m + 2, 4n + 2), h (4m + 3, 4n + 2), h (4m, 4n + 3), h (4m + 1, 4n + 3), h (4m + 2, 4n + 3), h (4m 3, 4n + 3) top left pixel h (4m of, 4n) is the case of the timing to be supplied to the arithmetic unit 23 only, the H level, in other cases have been made so as to be L level.

  The switch 22 selects the terminal 22a or 22b when the output of the NOR gate 25 is L level or H level.

  The reading unit 21 reads data (stored data) stored at an address corresponding to a signal output from the address supply circuit 1.

  Therefore, at the timing when the pixel h (4m, 4n) in the first hierarchy is supplied to the computing unit 23, the data stored at the address (m, n) in the third hierarchy memory 4 is read out by the readout unit 21. In this case, since the lower 2 bits ha0 and ha1 of the horizontal address HA and the lower 2 bits va0 and va1 of the vertical address VA are both 0, the output of the NOR gate 25 is H The switch 22 selects the terminal 22b.

  As a result, 0 is supplied to the computing unit 23 via the switch 22.

  In the computing unit 23, 0 and the pixel h (4m, 4n) of the first layer are added, and the added value (0 + h (4m, 4n)) is supplied to the writing unit 24. The writing unit 24 writes the output of the arithmetic unit 23 to the address corresponding to the signal output from the address supply circuit 1, that is, the address (m, n) of the third hierarchy memory 4.

  Next, at the timing when the pixel h (4m + 1, 4n) on the right side of the pixel h (4m, 4n) in the first hierarchy is supplied to the computing unit 23, the readout unit 21 again stores the third hierarchy memory 4 in the third hierarchy memory 4. Data stored in the address (m, n) (here, 0 + h (4m, 4n)) is read and supplied to the terminal 22a.

  On the other hand, in this case, the lower 2 bits ha0 or ha1 of the horizontal address HA are 1 or 0, respectively, and the lower 2 bits va0 and va1 of the vertical address VA are both 0. Therefore, the output of the NOR gate 25 is L level. Thus, the switch 22 selects the terminal 22a.

  As a result, the data (stored data) read in the reading unit 21 (here, 0 + h (4m, 4n)) is supplied to the computing unit 23 via the switch 22.

  In the computing unit 23, the data supplied via the switch 22 and the pixel h (4m + 1, 4n) of the first layer are added, and the added value (0 + h (4m, 4n) + h (4m + 1, 4n)) is obtained. , Supplied to the writing unit 24. The writing unit 24 writes the output of the arithmetic unit 23 to the address corresponding to the signal output from the address supply circuit 1, that is, the address (m, n) of the third hierarchy memory 4.

  Next, at the timing when the pixel h (4m + 2, 4n) on the right side of the pixel h (4m + 1, 4n) in the first hierarchy is supplied to the computing unit 23, the readout unit 21 again stores the third hierarchy memory 4 in the third hierarchy memory 4. Data stored in the address (m, n) (here, 0 + h (4m, 4n) + h (4m + 1, 4n)) is read and supplied to the terminal 22a.

  On the other hand, in this case, the lower 2 bits ha0 or ha1 of the horizontal address HA are 1 or 0, respectively, and the lower 2 bits va0 and va1 of the vertical address VA are both 0. Therefore, the output of the NOR gate 25 is L level. Thus, the switch 22 selects the terminal 22a.

  As a result, the data (stored data) read in the reading unit 21 (here, 0 + h (4m, 4n) + h (4m + 1, 4n)) is supplied to the computing unit 23 via the switch 22.

  In the arithmetic unit 23, the data supplied via the switch 22 and the pixel h (4m + 2, 4n) of the first layer are added, and the added value (0 + h (4m, 4n) + h (4m + 1, 4n) + h ( 4m + 2, 4n)) is supplied to the writing unit 24. The writing unit 24 writes the output of the arithmetic unit 23 to the address corresponding to the signal output from the address supply circuit 1, that is, the address (m, n) of the third hierarchy memory 4.

  Hereinafter, pixels h (4m + 3, 4n), h (4m, 4n + 1), h (4m + 1, 4n + 1), h (4m + 2, 4n + 1), h (4m + 3, 4n + 1), h (4m, 4n + 2), h in the first layer (4m + 1, 4n + 2), h (4m + 2, 4n + 2), h (4m + 3, 4n + 2), h (4m, 4n + 3), h (4m + 1, 4n + 3), h (4m + 2, 4n + 3), h (4m + 3, 4n + 3) In any case, the same processing as described above is performed at the timing supplied to the device 23, and as a result, the address (m, n) of the third hierarchical memory 4 is finally set to the above-described case. The pixel (pixel value) q (m, n) in the third hierarchy shown in Expression (9) is stored.

  As described above, the third hierarchical memory 4 stores the third hierarchical SD image in which one screen is composed of 128 × 128 pixels.

  Accordingly, the SD image data of the second hierarchy, and further the SD image data of the third hierarchy can be generated and stored at the same time as the SD image data of the first hierarchy is stored. That is, SD image data of the second and third hierarchies can be obtained in real time.

  Next, reading of the third hierarchy SD image from the third hierarchy memory 4 will be described.

  At the time of reading, when the hierarchy flag indicates the third hierarchy, the address supply circuit 1 again uses the upper 7 bits ha2 to ha8 or va2 to va8 of the horizontal address HA or the vertical address VA supplied thereto, respectively. The lower two bits ha0 and ha1 or va0 and va1 are output to the RMW circuit 5 as control signals.

  On the other hand, in the RMW circuit 5, the hierarchy flag, the R / W signal, and the output of the NOR gate 25 are supplied to the reading unit 21, and the reading unit 21 reads the R / W signal. When the hierarchy flag represents the third hierarchy, the third hierarchy stored in the address corresponding to the signal output from the address supply circuit 1 only when the output of the NOR gate 25 is at the H level. SD image data is read out and output.

  That is, from the above, the set of the horizontal address HA and the vertical address VA is (4m, 4n), (4m + 1, 4n), (4m + 2, 4n), (4m + 3, 4n), (4m, 4n + 1), ( 4m + 1, 4n + 1), (4m + 2, 4n + 1), (4m + 3, 4n + 1), (4m, 4n + 2), (4m + 1, 4n + 2), (4m + 2, 4n + 2), (4m + 3, 4n + 2), (4m, 4n + 3), (4m + 1, In the case of (4n + 3), (4m + 2, 4n + 3), (4m + 3, 4n + 3), the address supply circuit 1 outputs the same address (m, n). Therefore, simply reading out data from the address of the third hierarchical memory 4 corresponding to the signal output from the address supply circuit 1, the same data is read out 16 times.

  Therefore, in the reading unit 21, only when the set of the horizontal address HA and the vertical address VA is, for example, (4m, 4n), that is, only when the output of the NOR gate 25 is at the H level. The third layer pixel (pixel value) m (m, n) is read from the address (m, n) of the third layer memory 4.

  The third-layer SD image data read by the reading unit 21 is supplied to the switch 26. The switch 26 is turned on only when the R / W signal indicates reading, and is turned off in other cases. Therefore, in this case, the switch 26 is turned on. The read third-layer SD image data is output via the switch 26.

  As described above, the third hierarchy SD image for one screen composed of 128 × 128 pixels stored therein is read out from the third hierarchy memory 4. That is, the SD image of the 3rd hierarchy scanned sequentially is output by this.

  The reading of the second layer SD image data from the second layer memory 3 is performed as described above. For example, the lower 8 bits ha0 to ha7 of the horizontal address HA and the vertical address are read from the address supply circuit 1. It is also possible to perform by giving the lower 8 bits va0 to va7 of VA as the address of the second hierarchical memory 3. Similarly, when reading the third layer SD image data from the third layer memory 4, the address supply circuit 1 reads the lower 7 bits ha0 to ha6 of the horizontal address HA and the lower 7 bits va0 to va6 of the vertical address VA. This can be done by giving it as the address of the third hierarchical memory 4.

  As described above, the first hierarchy memory 101, the second hierarchy memory 102, or the third hierarchy memory 103 of FIG. 1 includes the first hierarchy memory 2, the second hierarchy memory 3, or the third hierarchy memory of FIG. The SD images of the first to third layers stored in 4 are stored. Accordingly, the first hierarchy memory 101, the second hierarchy memory 102, or the third hierarchy memory 103 of FIG. 1 is basically the first hierarchy memory 2, the second hierarchy memory 3, or the third hierarchy memory 4 of FIG. Each is configured similarly.

  Next, FIG. 7 shows a configuration example of the multi-hierarchy class classification unit 104 (multi-hierarchy class classification circuits 104a to 104d) in FIG.

The first to third hierarchy SD images respectively stored in the first hierarchy memory 101, the second hierarchy memory 102, or the third hierarchy memory 103 are supplied to the selection circuit 112. In addition, the SD image of the first or second hierarchy is also supplied to the activity detection circuit 111 ₁ or 111 ₂ , respectively.

For example, the activity detection circuit 111 ₁ or 111 ₂ detects the activity using the SD pixel near the position of the target pixel in the SD image of the first or second hierarchy, and the detection result is used as the selection circuit 112. Has been made to detect. The selection circuit 112 selects any one of the first to third layer SD images based on the activities from the activity detection circuits 111 ₁ and 111 ₂ and supplies the selected SD image to the class tap generation circuit 113. Yes. In addition, the selection circuit 112 is configured to output, for example, a 2-bit selection signal to the synthesis circuit 115 indicating which one of the first to third layers is selected. The class tap generation circuit 113 generates a class tap for classifying the target pixel using the image from the selection circuit 112 and supplies the class tap to the class classification circuit 114. The class classification circuit 114 performs class classification using the class tap from the class tap generation circuit 113 and supplies the class classification result to the synthesis circuit 115. The synthesis circuit 115 synthesizes the selection signal from the selection circuit 112 and the class as the class classification result from the class classification circuit 114 into one value, which is used as the final class classification result of the pixel of interest. The adaptive processing unit 105 (FIG. 1) is supplied.

  Next, the operation will be described.

First, the activity detection circuit 111 ₁ or 111 ₂ detects an activity near the target pixel in the SD image of the first or second hierarchy.

In other words, the activity detection circuit 111 ₁ detects an activity in a range of 3 × 3 pixels centering on, for example, a pixel that coincides with the position of the target pixel among the pixels constituting the SD image of the first hierarchy. Also, the activity detection circuit 111 ₂ detects an activity in a 3 × 3 pixel range centering on, for example, a pixel that coincides with the position of the target pixel among the pixels constituting the SD image of the second hierarchy.

Here, since the SD image in the second layer is an image in which the number of horizontal or vertical pixels of the SD image in the first layer is halved, the activity detection can be performed based on the SD image in the first layer. The circuit 111 ₂ detects an activity in a wider range than the range in the activity detection circuit 111 ₁ .

  As described above, a 3 × 3 pixel range centering on the first and second layer SD pixels that coincides with the position of the target pixel, that is, a square shape corresponding to the class tap described in FIG. The activity in the range is detected in, for example, the multi-layer class classification circuit 104a, and the other multi-layer class classification circuits 104b to 104d correspond to, for example, the class taps described in FIGS. Each activity in the range is detected.

Any activity on the first or second layer image detected by the activity detection circuit 111 ₁ or 111 ₂ is supplied to the selection circuit 112. In the selection circuit 112, _{one of} the first to third hierarchy SD images is selected based on the activities from the activity detection circuits 111 ₁ and 111 ₂ .

  That is, the selection circuit 112 determines whether or not the activity for the image in the first layer is greater than the predetermined threshold ε. When the activity for the first layer image is larger than the predetermined threshold ε, the selection circuit 112 selects the first layer image and supplies it to the class tap generation circuit 113.

  When the activity for the first layer image is not greater than the predetermined threshold ε, the selection circuit 112 determines whether the activity for the second layer image is greater than the predetermined threshold ε. When the activity for the second layer image is larger than the predetermined threshold ε, the selection circuit 112 selects the second layer image and supplies it to the class tap generation circuit 113.

  If the activity for the second layer image is not greater than the predetermined threshold ε, the selection circuit 112 selects the third layer image and supplies it to the class tap generation circuit 113.

  Further, the selection circuit 112 supplies a selection signal representing the selected hierarchy to the synthesis circuit 115.

  In the class tap generation circuit 113, a class tap is generated (formed) using the hierarchical image supplied from the selection circuit 112.

  That is, when the first level SD image is selected by the selection circuit 112, that is, the SD corresponding to the position of the target pixel among the SD pixels constituting the first level SD image indicated by R1 in FIG. When the activity in the range of 3 × 3 pixels centered on the pixel is larger than the threshold ε, the class tap generation circuit 113 sets the 3 × 3 first-layer SD pixel as a class tap, and the class classification circuit 114 To be supplied. In FIG. 8, the ◯ marks indicate the first-level SD pixels, and the X marks indicate the HD pixels.

  Further, when the SD image of the second hierarchy is selected by the selection circuit 112, that is, based on the SD image of the first hierarchy, the SD image of the second hierarchy (R1 in FIG. 8) is indicated by R2. Among the SD pixels constituting the SD image (the image in which the number of horizontal or vertical pixels is halved), the activity in the range of 3 × 3 pixels centering on the SD pixel that matches the position of the target pixel is the threshold value. If it is greater than ε, the class tap generation circuit 113 sets the 3 × 3 second-layer SD pixel as a class tap and supplies it to the class classification circuit 114. Accordingly, in this case, if the SD image in the first layer is used as a reference, nine SD pixels in the second layer in the range in which the horizontal or vertical range is four times when the SD image in the first layer is selected. A class tap is formed.

  Further, when the SD image of the third hierarchy is selected by the selection circuit 112, the class tap generation circuit 113 uses the SD image of the first hierarchy as a reference, and the SD image of the third hierarchy indicated by R3 in FIG. Of the SD pixels constituting (the image in which the number of horizontal or vertical pixels of the SD image in the first layer is ¼), 3 × 3 pixels centered on the SD pixel that matches the position of the target pixel are The class tap is supplied to the class classification circuit 114. Accordingly, in this case, if the SD image of the first hierarchy is used as a reference, nine SD pixels of the third hierarchy in the range of 16 times the horizontal or vertical range when the SD image of the first hierarchy is selected. A class tap is formed.

  As described above, the 3 × 3 pixel range centered on the first to third layer SD pixels that coincides with the position of the target pixel, that is, the class based on the SD pixels in the positional relationship as described in FIG. The tap is formed in, for example, the multi-layer class classification circuit 104a, and in the other multi-layer class classification circuits 104b to 104d, for example, the pixels having the positional relationship described with reference to FIGS. 14 to 16 are used. Each class tap is formed.

  The class classification circuit 114 classifies the pixel of interest using the class tap from the class tap generation circuit 113 in the same manner as in the class classification circuit 203 in FIG. The classification result is supplied to the synthesis circuit 115. In the synthesis circuit 115, for example, the selection signal from the selection circuit 112 is added as the high-order bits of the value as the class classification result from the class classification circuit 114, and this is the final class of the pixel of interest (the value representing the class). ) To the adaptive processing unit 105.

  As described above, with reference to the SD image of the first layer, the target pixel is classified using the image of the layer corresponding to the range in which the activity has a certain size (here, the threshold value ε). Since the class tap is formed, it is possible to obtain a class corresponding to the pixel of interest by performing class classification using the class tap.

  In the above description, the activity of the images in the first and second layers is detected. However, it is also possible to detect the activity in the second and third layers. In this case, for example, when the activity of the third hierarchy is smaller than the predetermined threshold ε ′, the class tap may be formed using the image of the third hierarchy. If the activity of the third hierarchy is equal to or higher than the predetermined threshold ε ′, but the activity of the second hierarchy is smaller than the predetermined threshold ε ′, a class tap is formed using the image of the second hierarchy. You can do it. Furthermore, when the activity of the second layer is equal to or higher than the predetermined threshold ε ′, the class tap may be formed using the image of the first layer.

  Next, FIG. 9 shows a configuration example of the adaptive processing unit 105 (each of the adaptive processing circuits 105a to 105d) in FIG.

  The prediction tap generation circuit 121, the prediction calculation circuit 122, or the coefficient ROM 123 is configured similarly to the prediction tap generation circuit 205, the prediction calculation circuit 206, or the coefficient ROM 207 in FIG.

  That is, in the prediction tap generation circuit 121, the target pixel used for obtaining the predicted value of the target pixel in the prediction calculation circuit 122 from the SD image of the first layer supplied from the first layer memory 101 (FIG. 1). On the other hand, a plurality of SD pixels having a predetermined positional relationship are extracted and supplied to the prediction calculation circuit 122 as prediction taps.

  Specifically, here, for example, prediction taps as described with reference to FIGS. 13 to 16 are formed (in the prediction tap generation circuit 121 in each of the adaptive processing circuits 105 a to 105 d) and supplied to the prediction calculation circuit 122. .

  In addition to the prediction tap supplied from the prediction tap generation circuit 121, the prediction arithmetic circuit 122 is also supplied with a prediction coefficient from the coefficient ROM 123.

  That is, the coefficient ROM 123 stores the prediction coefficient obtained by learning in advance for each class, and the address terminal (AD) has a multilevel class classification unit 104 (FIG. 1). The class of the pixel of interest is supplied. When the class is supplied from the multi-layer class classification unit 104, the coefficient ROM 123 reads the prediction coefficient stored at the address corresponding to the class and supplies the prediction coefficient to the prediction calculation circuit 122.

As a result, the prediction calculation circuit 122 is supplied with the prediction tap corresponding to the target pixel and the prediction coefficient for the class of the target pixel. In the prediction arithmetic circuit 122, the prediction coefficients w, w ₂ ,... From the coefficient ROM 123, and the prediction taps (constituting SD pixels) x ₁ , x ₂ ,. Is used to calculate the prediction value E [y] of the pixel of interest (HD pixel) y, and this is output as the pixel value of the HD pixel.

  As described above, since a more appropriate class is supplied to the target pixel from the multi-layer class classification unit 104, the result is obtained by performing the above-described adaptive processing corresponding to the class. It is possible to improve the image quality of HD images.

  Next, FIG. 10 illustrates another configuration example of the multi-layer class classification unit 104 (multi-layer class classification circuits 104a to 104d) in FIG.

The first to third layer image data stored in the first layer memory 101, the second layer memory 102, or the third layer memory 103 are supplied to the class tap generation circuits 131 _{1 to} 131 ₃ , respectively. Has been made.

For example, as in the case of the class tap generation circuit 113 in FIG. 7, the class tap generation circuits 131 _{1 to} 131 ₃ use class taps for classifying the target pixel using the first to third layer images. Generated (formed) and supplied to the classifying circuits 132 _{1 to} 132 ₃ , respectively. The class classification circuits 132 _{1 to} 132 ₃ use the class taps from the class tap generation circuits 131 _{1 to} 131 ₃ , for example, classify the target pixel in the same manner as in the class classification circuit 114 of FIG. The classes are supplied to the synthesis circuit 133, respectively. The combining circuit 133 combines the classes from the class classification circuits 132 _{1 to} 132 ₃ and supplies the combined result to the adaptive processing unit 105 (FIG. 1) as the final class of the pixel of interest. Yes.

In the multi-hierarchy class classification unit 104 (multi-hierarchy class classification circuits 104a to 104d) configured as described above, in the class tap generation circuits 131 _{1 to} 131 ₃ , the class taps display the first to third hierarchy images. Respectively.

That is, in the class tap generation circuits 131 _{1 to} 131 ₃ , for example, when the first to third hierarchy images are selected in the selection circuit 112 of FIG. The same class taps are formed using the first to third layer images. The class taps formed using the first to third layer images are supplied to the class classification circuits 132 _{1 to} 132 ₃ , respectively.

In the class classification circuits 132 _{1 to} 132 ₃ , class classification is performed using the class taps from the class tap generation circuits 131 _{1 to} 131 ₃ , and three classes (first to third hierarchies) of the pixel of interest obtained as a result are obtained. All of the classification results using class taps formed from the respective images are supplied to the synthesis circuit 133.

In the synthesis circuit 133, the classes from each of the class classification circuits 132 _{1 to} 132 ₃ are synthesized into one. That is, for example, the synthesis circuit 133 sets the values representing the classes from the class classification circuits 132 _{1 to} 132 ₃ into one bit string as the upper, middle, and lower bits, respectively. This value is supplied to the adaptive processing unit 105 as the final class of the pixel of interest.

  As described above, the classification is performed using the SD images of the first to third layers, and the result of combining the classification results is used as the final classification result of the pixel of interest. Equivalently, since the classification is performed using not only the SD pixel near the target pixel but also the SD pixel separated from the target pixel, the target pixel is matched with the target pixel. Can be classified into different classes. Furthermore, by performing adaptive processing corresponding to the class, it is possible to improve the image quality of the resulting HD image.

  In FIG. 8, for example, it is expected that the same effect as in the above case can be obtained by classifying using all SD pixels in the first layer in the range indicated by R3 as class taps. In this case, since the number of SD pixels in the first layer in the range R3 is large, the processing load becomes enormous.

That is, for example, in the embodiment of FIG. 7, the class tap formed in the class tap generation circuit 113 is configured by nine SD pixels as in the case of the class tap generation circuit 202 of FIG. Since a 2-bit selection signal is added to the classification result using the class tap, simply classifying it into one of the number of classes proportional to 2 ¹¹ (= 2 ⁹ × 2 ² ). Is done.

In the embodiment of FIG. 10, each of the class tap generation circuits 113 _{1 to} 113 ₃ has a class tap composed of nine SD pixels as in the class tap generation circuit 202 of FIG. After forming and classifying using the three class taps, the three class classification results are combined, so it is simply proportional to 2 ²⁷ (= 2 ⁹ × 2 ⁹ × 2 ⁹ ) Classification is performed on any of the determined number of classes.

On the other hand, the range R3 in FIG. 8 includes 289 (= 17 × 17) first-layer SD pixels, so that when a class tap is formed by this, it is simply 2 ^289. It is necessary to classify one of the classes in proportion to the enormous value. Therefore, in this case, the processing burden is enormous.

  Next, FIG. 11 shows a configuration example of a learning apparatus that performs a learning process for calculating a prediction coefficient to be stored in the coefficient ROM 123 of FIG. In the figure, portions corresponding to those in FIG. 17 are denoted by the same reference numerals.

The HD image to be the teacher data y in learning is supplied to the thinning circuit 141 ₁ and the teacher data extracting circuit 146. In the thinning circuit 141 ₁ , for example, as in the storage device of FIG. (In the storage device of FIG. 2, in the same manner as when the SD image of the second layer is generated from the SD image of the first layer), the number of horizontal or vertical pixels of the HD image is halved. A one-layer SD image is constructed. The SD image of the first layer is supplied to the thinning circuit 141 ₂ , the multi-layer class classification unit 144, and the prediction tap generation circuit 145.

Any thinning circuit 141 _2, in the same manner as in thinning circuit 141 _1, SD image of the second hierarchy horizontal or vertical number of pixels each of the first hierarchical SD image is 1/2 is constructed. The SD image of the second layer is supplied to the thinning circuit 141 ₃ and the multi-layer class classification unit 144. Similarly to the case of the thinning circuit 141 ₁ , the thinning circuit 141 ₃ also forms a _third hierarchical SD image in which the number of horizontal or vertical pixels of the second hierarchical SD image is halved. This third-layer SD image is supplied to the multi-layer class classification unit 144.

  The multi-hierarchy class classifying unit 144 is configured in the same manner as the multi-hierarchy class classifying unit 104 shown in FIG. 7 or 10, and uses the first to third hierarchies images supplied to the class, as described above. Classification (multi-layer classification) is performed. The class as the classification result is supplied to the prediction tap memory 147 and the address terminal (AD) of the teacher data memory 148.

Further, in the prediction tap generation circuit 145, the same processing as in the prediction tap generation circuit 121 in FIG. 9 is performed, whereby the prediction tap for obtaining the prediction value of the target pixel is the _first from the thinning-out circuit 141 ₁ . It is formed using hierarchical SD images. This prediction tap is supplied to the prediction tap memory 147.

  In the prediction tap memory 147, the prediction tap supplied from the prediction tap generation circuit 145 is stored at the address corresponding to the class supplied from the multi-layer class classification unit 144.

  On the other hand, in the teacher data extraction circuit 146, the HD pixel which is the target pixel in the multi-layer class classification unit 144 and the prediction tap generation circuit 145 is extracted from the HD image supplied thereto, and the teacher data memory 148 is used as teacher data. To be supplied.

  The teacher data memory 148 stores the teacher data supplied from the teacher data extraction circuit 146 at the address corresponding to the class supplied from the multilevel class classifying unit 144.

  The above processing is sequentially performed on all HD pixels constituting all HD images prepared for learning in advance as the target pixel.

  As a result, at the same address in the teacher data memory 148 or the prediction tap memory 147, the HD pixel of the class corresponding to the address, or the position that constitutes the prediction tap described in FIGS. A certain SD pixel is stored as teacher data y or learning data x, respectively.

  Thereafter, the arithmetic circuit 149 reads prediction pixels as learning data or HD pixels as teacher data stored in the same address from the prediction tap memory 147 or the teacher data memory 148, and uses them, for example, the minimum self A prediction coefficient that minimizes an error between the predicted value and the teacher data is calculated by multiplication. That is, in the arithmetic circuit 149, the normal equation shown in Expression (7) is established for each class, and the prediction coefficient is obtained by solving this.

  As described above, the prediction coefficient for each class obtained by the arithmetic circuit 149 is stored in the address corresponding to the class in the coefficient ROM 123 of FIG.

  In the embodiment of FIG. 7, the selection circuit 112 selects any one of the first to third hierarchy SD images, and the class tap generation circuit 113 uses the image of the first hierarchy. However, for example, the selection circuit 112 selects two of the SD images of the first to third hierarchies, and the class tap generation circuit 113 displays the two class taps. Class taps can be formed using each hierarchy. In this case, the class classification circuit 114 performs class classification for each of the class taps formed using the two hierarchies, and the synthesis circuit 115 combines the two class classification results into one. good.

  Further, in the present embodiment, the prediction tap is formed using the first layer image having the largest number of pixels among the first to third layer SD images. It is also possible to form the image using two or more of the third layer images.

  Further, in the present embodiment, in the storage device of FIG. 2, the second hierarchical memory 3 and the third hierarchical memory 4 are given a part of the horizontal address HA and the vertical address VA given to the first hierarchical memory 2. However, in addition to the horizontal address HA and the vertical address VA given to the first hierarchy memory 2, dedicated (independent) addresses are given to the second hierarchy memory 3 and the third hierarchy memory 4. It is also possible to access.

  Further, in the present embodiment, in the storage device of FIG. 2, the pixels constituting the SD image are positioned in the horizontal or vertical direction with respect to the first hierarchy memory 2, the second hierarchy memory 3, and the third hierarchy memory 4. Each of the first hierarchical memory 2, the second hierarchical memory 3 and the third hierarchical memory 4 is further accessed with, for example, an address corresponding to the time direction. It is also possible to give access. In this case, the second and third layer pixels are formed by adding the first layer pixels scattered in the time direction in addition to the first layer pixels scattered in the horizontal and vertical spatial directions.

  Similarly, class taps and prediction taps can also be formed using SD pixels scattered not only in the spatial direction but also in the temporal direction.

  Further, the first hierarchy memory 101, the second hierarchy memory 102, and the third hierarchy memory 103 in FIG. 1, and the first hierarchy memory 2, the second hierarchy memory 3, and the third hierarchy memory 4 in FIG. It is not necessary to have a single memory, and it is possible to configure all of them with a single memory. In this case, the storage area of one memory is divided into three memory areas of the first hierarchy memory 101, the second hierarchy memory 102, and the third hierarchy memory 103, the first hierarchy memory 2, the second hierarchy memory 3, and the third hierarchy memory, respectively. 4 may be assigned to each of the three.

  Further, in the present embodiment, in FIG. 2, all of the address supply circuit 1, the first hierarchy memory 2, the second hierarchy memory 3, the third hierarchy memory 4, and the RMW circuit 5 are formed on one chip. However, these are not necessarily formed on one chip.

  Furthermore, in the present embodiment, the bit allocation amount of the pixels in the first hierarchy is 8 bits, and the data length of the memory cells in the first hierarchy memory 2, the second hierarchy memory 3, or the third hierarchy memory 4 is set to the first Or 8 bits, 10 bits, or 12 bits so as not to drop the pixels in the third layer, but the data lengths of the memory cells of the first layer memory 2, the second layer memory 3, and the third layer memory 4 For example, it may be uniformly 8 bits. However, in this case, for pixels in the second or third hierarchy, a value obtained by rounding down the lower 2 bits of the addition value of the 2 × 2 pixels in the first or second hierarchy (this value is obtained by dividing the addition value by 4). Therefore, the loss of data will be lost.

  The present invention is applicable to both non-interlaced scanned images and interlaced scanned images.

  Furthermore, in the present embodiment, the number of hierarchies of the SD image is three, but the number of hierarchies may be two, or four or more.

  Further, in the present embodiment, the addition value of 4 SD pixels of 2 × 2 in the lower hierarchy is set as the upper hierarchy SD pixel (pixel value) one level higher. The method of forming is not limited to this.

  Furthermore, the present invention can be realized by hardware and also by causing a computer to execute a program that performs the above-described processing.

  In this embodiment, pixels (pixel values) are stored in a memory represented by, for example, a RAM (Random Access Memory). However, the pixels may be stored in, for example, a magnetic disk or a magneto-optical device. It is also possible to store (record) in a recording medium such as a disk, a magnetic tape, or an optical card.

  In addition, the image processing apparatus in FIG. 1 and the storage apparatus in FIG. 2 can be configured as separate devices or in an integrated manner.

  Further, the present invention can be applied not only when an SD image is converted into an HD image but also when an image is enlarged, for example.

  Furthermore, the positional relationship of the SD pixels constituting the class tap and the prediction tap is not limited to the above.

It is a block diagram which shows the structural example of one Embodiment of the image processing apparatus to which this invention is applied. It is a block diagram which shows the structural example of the memory | storage device which produces | generates the 1st thru | or 3rd hierarchy SD image. It is a figure which shows the structural example of 1 screen of the SD image of a 1st hierarchy. FIG. 3 is a block diagram illustrating a first functional configuration example of the storage device of FIG. 2. FIG. 3 is a block diagram illustrating a second functional configuration example of the storage device of FIG. 2. FIG. 4 is a block diagram illustrating a third functional configuration example of the storage device in FIG. 2. FIG. 2 is a block diagram illustrating a first configuration example of each of multi-layer class classification circuits 104a to 104d in FIG. It is a figure for demonstrating the class tap which the class tap generation circuit 113 of FIG. 7 produces | generates. FIG. 2 is a block diagram illustrating a configuration example of each of adaptive processing circuits 105a to 105d in FIG. 1. FIG. 3 is a block diagram illustrating a second configuration example of each of the multilevel class classification circuits 104a to 104d in FIG. 1. It is a block diagram which shows the structural example of one Embodiment of the learning apparatus which learns the prediction coefficient memorize | stored in coefficient ROM123 of FIG. It is a block diagram which shows the structure of an example of the conventional image converter. It is a figure for demonstrating the formation method of a class tap and a prediction tap. It is a figure for demonstrating the formation method of a class tap and a prediction tap. It is a figure for demonstrating the formation method of a class tap and a prediction tap. It is a figure for demonstrating the formation method of a class tap and a prediction tap. It is a block diagram which shows the structure of an example of the learning apparatus which learns the prediction coefficient memorize | stored in coefficient ROM207 of FIG.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Address supply circuit, 2 1st hierarchy memory, 3 2nd hierarchy memory, 4 3rd hierarchy memory, 5 RMW circuit, 11 reading part, 12 switch, 12a, 12b terminal, 13 calculator, 14 writing part, 15 NOR gate , 16 switch, 21 reading unit, 22 switch, 22a, 22b terminal, 23 arithmetic unit, 24 writing unit, 25 NOR gate, 26 switch, 101 first layer memory, 102 second layer memory, 103 third layer memory, 104 Multi-layer class classification unit, 104a to 104d multi-layer class classification circuit, 105 adaptive processing unit, 105a to 105d adaptive processing circuit, 106 HD image memory, 106a to 106d memory, 111 ₁ , 111 ₂ activity detection circuit, 112 selection circuit, 113 Class tap generation times , 114 classifying circuit, 115 combining circuit, 121 the prediction tap generating circuit, 122 predictive operation circuit, 123 coefficients ROM, 131 ₁ to 131 ₃ class-tap generating circuit, 132 ₁ to 132 ₃ classification circuit 133 synthesizing circuit, 141 ₁ Thru | or 141 ₃ thinning circuit, 144 multi-layer class classification | category part, 145 prediction tap production | generation circuit, 146 teacher data extraction circuit, 147 prediction tap memory, 148 teacher data memory, 149 arithmetic circuit

Claims (8)

A second image in which the first image is represented by a number of pixels smaller than the number of pixels in the first image, and a third image in which the number of pixels is smaller than the number of pixels in the second image. An image processing apparatus for performing a process for obtaining the first image,
Activity detection means for detecting a part of the second and third images corresponding to the target pixel in the second and third images, with the pixel of the first image being focused on as the target pixel;
A selection means for selecting an image having an activity exceeding a predetermined threshold from the second and third images;
Class classification that classifies the pixel of interest into a predetermined class by setting a value corresponding to a pixel value pattern of a pixel in a predetermined region centered on a pixel of the selected image corresponding to the pixel of interest as a class Classification means for performing
Predicting means for predicting the predicted value of the target pixel corresponding to the class of the target pixel;
The prediction means includes
Prediction coefficient storage means for storing, for each class, a prediction coefficient for calculating a prediction value of the target pixel by linear combination with the pixels of the second and third images;
By linearly combining the prediction coefficient for the class of the target pixel and pixel values of a plurality of pixels having a predetermined positional relationship with respect to the target pixel among the pixels of the second and third images. And an estimated value calculating means for obtaining a predicted value of the target pixel. The value corresponding to the pixel value pattern of the pixels in the predetermined area of the selected image is obtained by performing ADRC processing on the pixel values of the pixels in the predetermined area. Image processing device. The second and third images have a smaller number of pixels than the first image by adding pixel values of adjacent pixels of the first image to a pixel value of one pixel. The image processing apparatus according to claim 1, wherein the image processing apparatus is an image. A second image in which the first image is represented by a number of pixels smaller than the number of pixels in the first image, and a third image in which the number of pixels is smaller than the number of pixels in the second image. Using an image processing method for performing a process for obtaining the first image,
A pixel of the first image of interest is a pixel of interest, and an activity of a portion corresponding to the pixel of interest in the second and third images is detected;
Selecting an image having an activity exceeding a predetermined threshold from the second and third images;
Classifying the pixel of interest into a predetermined class by classifying a value corresponding to a pixel value pattern of a pixel in a predetermined region centered on a pixel of the selected image corresponding to the pixel of interest;
Predicting the predicted value of the target pixel corresponding to the class of the target pixel;
When predicting the predicted value of the target pixel, a prediction coefficient that stores, for each class, a prediction coefficient for calculating the predicted value of the target pixel by linear combination with the pixels of the second and third images The prediction coefficient for the class of pixel of interest acquired from the storage means, and pixel values of a plurality of pixels having a predetermined positional relationship with respect to the pixel of interest among the pixels of the second and third images An image processing method characterized by obtaining a predicted value of the target pixel by linearly combining. A second image representing a first image with a number of pixels less than the number of pixels of the first image; a third image representing a number of pixels less than the number of pixels of the second image; An image processing apparatus that performs processing for obtaining the first image using a fourth image represented by a smaller number of pixels than the number of pixels of the third image,
Activity detecting means for detecting a part of the second image to the fourth image corresponding to the target pixel in the second to fourth images, using the first image pixel of interest as the target pixel;
Selecting means for selecting a plurality of images having an activity exceeding a predetermined threshold from the second to fourth images;
Each of the values corresponding to the pixel value pattern of the pixels in the predetermined area centered on the selected pixels of the plurality of images corresponding to the target pixel is obtained and converted into one bit string. Class classification means for classifying into a predetermined class represented by the bit string;
Predicting means for predicting the predicted value of the target pixel corresponding to the class of the target pixel;
The prediction means includes
Prediction coefficient storage means for storing, for each class, a prediction coefficient for calculating a prediction value of the target pixel by linear combination with the pixels of the second to fourth images;
By linearly combining the prediction coefficient for the class of the target pixel and pixel values of a plurality of pixels having a predetermined positional relationship with respect to the target pixel among the pixels of the second to fourth images. And an estimated value calculating means for obtaining a predicted value of the target pixel. The value corresponding to the pixel value pattern of the pixels in the predetermined area of the selected plurality of images is obtained by ADRC processing the pixel values of the pixels in the predetermined area. The image processing apparatus described. The second to fourth images have a smaller number of pixels than the first image by adding pixel values of adjacent pixels of the first image to a pixel value of one pixel. The image processing apparatus according to claim 5, wherein the image processing apparatus is an image. A second image representing a first image with a number of pixels less than the number of pixels of the first image; a third image representing a number of pixels less than the number of pixels of the second image; An image processing method for performing processing for obtaining the first image using a fourth image represented by a smaller number of pixels than the number of pixels of the third image,
A pixel of the first image of interest is a pixel of interest, and an activity of a portion corresponding to the pixel of interest in the second to fourth images is detected;
Selecting a plurality of images having an activity exceeding a predetermined threshold from the second to fourth images;
Each of the values corresponding to the pixel value pattern of the pixels in the predetermined area centered on the selected pixels of the plurality of images corresponding to the target pixel is obtained and converted into one bit string. Classify into a predetermined class represented by the bit string,
Predicting the predicted value of the target pixel corresponding to the class of the target pixel;
When predicting the predicted value of the pixel of interest, a prediction coefficient storing a prediction coefficient for calculating the predicted value of the pixel of interest by linear combination with the pixels of the second to fourth images for each class The prediction coefficient for the class of pixel of interest acquired from the storage means, and pixel values of a plurality of pixels having a predetermined positional relationship with respect to the pixel of interest among the pixels of the second to fourth images An image processing method characterized by obtaining a predicted value of the target pixel by linearly combining.

Images