JP4144091B2 - Image processing apparatus and method - Google Patents

Description

【００１８】
なお、式（２）における値ｍは図３（Ａ）の例の場合、３６８となる。求められた係数をｃ１’乃至ｃ４’とするとき、式（３）に示す平面式が生成される。
ｒ＝ｃ１’・ｘ＋ｃ２’・ｙ＋ｃ３’・ｚ＋ｃ４’ ・・・（３）
【００１９】
このようにして、平面推定部３により求められた係数ｃ１’乃至ｃ４’を用いて表される式（３）により生成される平面が推定平面とされる。この推定平面の一例が図４に示されている。図４において示された推定平面は、式（３）により生成される平面であり、局所的時空間ブロック内に実際に存在する画素により生成される平面、換言すれば、推定平面上に局所的時空間ブロックの画素が乗っている平面とは限らない。さらに換言すると、推定平面は、信号レベル（輝度値）がほぼ同一であると判断される画素が存在するであろう位置に存在する平面である。
【００２０】
図４に示したような推定平面が推定される場合、すなわち、垂直方向と水平方向からなる面の水平方向においては、左側から右側にかけて垂直座標値が下がり、時間方向においては、水平と垂直の座標値共に変化がない平面が推定される場合、図中右上側から左下側にかけて、徐々に輝度値が下がる、或いは上がる（グラデーションがかかっているような）画像が、９フィールド分の時刻の間、変化なく表示されていることになる。
【００２１】
なお、図４に示した推定平面は、等レベル面を表現しているものであり、信号のレベルそのものを表現しているものではなく、信号のレベル値の等高線のようなものとして示してある。
【００２２】
図４に示したような推定平面が推定される他の例としては、先の空間方向に関し徐々に輝度値が変化するものの他、階段上に急峻に変化する場合など、推定平面と直交する方向に何らかの輝度値変化を呈するような画像である。
【００２３】
次に、定常性方向評価部４は、このようにして、平面推定部３により求められた係数ｃ１’乃至ｃ４’を用いて表される推定平面に含まれ、少なくとも１次元方向に関し、傾斜が０の方向（定常方向）を求める。求められる傾斜０の方向は、１次元または２次元の関数、例えば、ｆ（ｘ）やｆ（ｘ，ｙ）で表現される直線式となる。求められる定常方向は、上述した輝度値をそのままプロットしたときに、階段状になる画像の場合、その階段のステップの方向、換言すれば、階段を上り下りする方向と直交する方向となる。
【００２４】
定常性方向評価部４は、定常方向を求めるとともに、クラス分類も行う。換言すると、クラスに基づく定常方向を判定する。すなわち、予め複数のクラスと、各クラスに分類される為の条件が定められており、どのクラスに分類されるかにより定常性が判断される。各クラスに分類される為の条件としては、９フィールドに含まれる３６９画素の内の所定の５画素の存在位置による。
【００２５】
この場合、総クラス数は、₃₆₉Ｃ₅個のクラスとなる。しかしながら、₃₆₉Ｃ₅個のクラスは膨大な組み合わせ数になるため実用的ではない。そこで、図５に示したように、５枚の第１フィールドの所定の２１画素（以下、適宜、選択候補画素と記述する）から５画素を用いてクラス数を考えると、₂₁Ｃ₅＝２０３４９個のクラスとなり、取り扱いやすいクラス数になる。さらに、注目画素を必ず含むという条件を付加することにより、選択候補画素の２０画素から４画素を選択することになるので、₂₀Ｃ₄＝４８４５個のクラスとなり、より取り扱いやすいクラス数となる。
【００２６】
図５に示した例では、基準フィールド内の選択候補画素の１７画素は、注目画素を含む縦横斜め方向に関する全ての方向において、５画素が選択できるように配置されている。そして、基準フィールドの前後のフィールドでは、注目画素と時間軸の座標値のみが異なる４点、換言すれば、注目画素と垂直方向と水平方向の座標値が同じ４点が選択候補画素とされている。
【００２７】
なお、図５の例においては、第１フィールドに存在する画素のみを用いているが、第２フィールドに存在する画素を選択候補画素として用いても良いし、２１画素以上の画素を選択候補画素としても良い。すなわち、選択候補画素としては、９フィールド内の全ての画素である３９６画素とすることが一番良いが、上述したように実用的ではないので、実用的な数で、なるべく多くの画素を選択候補画素とすることが望ましい。
【００２８】
注目画素を含む２１画素の選択候補画素を用いてクラスを作成すると、４８４５個のクラスが作成できる。図６に、４８４５個のクラスのうち、クラス０とクラス４８４４（一番最初と最後のクラス）、並びに特徴的なクラスであるクラス１５２とクラス２０８８が例として示されている。クラス番号の付け方は、スキャン順に基づいている。例えば、クラス０は、選択候補画素の２１画素のうち、最初にスキャンされる座標値（０，０，−４）の画素、その次にスキャンされる座標値（０，０，−２）の画素、さらにその次にスキャンされる座標値（−４，−４，０）の画素、そして、その次にスキャンされる座標値（０，−４，０）の画素、および座標値（０，０，０）の注目画素の合計５画素である。
【００２９】
同様にして、スキャン順に基づいてクラス番号を付けることにより、クラス０乃至４８４４が生成される。クラス１５２は、例えば静止画などのように、定常方向が時間軸方向に最も定常性が得られる画像が分類されるクラスである。クラス２０８８は、例えば、白地に斜めに１本の線が引いてあるような画像のように、同一フィールド内の斜め方向に最も定常性が得られる画像が分類されるクラスである。
【００３０】
定常性方向評価部４は、まず平面推定部３により求められた推定平面の式（３）に、順次、クラス０乃至４８４４内の５画素の座標値を代入し、その５画素毎に算出される値ｒの最小値と最大値の差（ダイナミックレンジ）をとる。そして、クラス毎に得られたダイナミックレンジのうち、最小のダイナミックレンジを有するクラスを、処理している局所的時空間ブロックの分類クラスとする。ただし、このようして定常性方向評価部４において算出される値ｒは、輝度値を意味するものではなく、単にダイナミックレンジを算出するための値として用いられている。
【００３１】
詳細に説明するに、まず、クラス０の５画素の座標値、すなわち、（０，０，−４），（０，０，−２），（−４，−４，０），（０，−４，０），（０，０，０）を、順次、式（３）に代入することにより、５つの値ｒが得られる。得られた５つの値ｒの最小値と最大値の差を取ることにより、クラス０のダイナミックレンジが得られる。同様の処理をクラス１乃至４８４４に対しても行うことにより、合計４８４５個のダイナミックレンジが得られる。これら４８４５個のダイナミックレンジのうち、最小のダイナミックレンジを有するクラスを、処理している画素のクラス（以下、分類クラスと称する）として決定する。仮に、ダイナミックレンジが０であるクラスが存在する場合、そのクラスの５画素は、式（３）で表される推定平面上に存在することを意味する。
【００３２】
このようにして、求められた分類クラスは、予測タップ構成部５と予測係数ROM６に供給される。ステップＳ５において、予測タップ構成部５は、供給された分類クラスに対応する予測タップの画素データを、信号遅延部２から読み出し、予測処理部７に出力する。また、予測係数ROM６は、供給された分類クラスに対応する予測係数を予測処理部７に出力する。予測タップとしては、図６に示したように、定常方向を判断するのに用いたクラス毎に設定された５画素が用いられる。従って、予測タップ構成部５が信号遅延部２から読み出す画素データは、分類クラスの５画素の座標値に対応する位置に存在する画素データ（輝度値）である。
【００３３】
予測処理部７は、ステップＳ６において、供給された予測係数と画素データを用いて適応処理し、処理後の画素データを出力する。なお、適応処理とは、注目画素のクラスに対応した予測係数と予測タップの画素データを用いて後述する式（４）に示す線形１次結合モデルの演算を行う処理のことである。
【００３４】
上述したように、画像処理装置においては、入力ＳＤ画像が、クラス毎に、予め予測係数ROM６に記憶された予測係数を用いて適応処理される。ここで、予測係数ROM６に記憶される予測係数を生成する予測係数学習装置について説明する。
【００３５】
図７は、予測係数学習装置の構成を示すブロック図である。そのブロック構成部２１、信号遅延部２２、平面推定部２３、定常性方向評価部２４、および予測タップ構成部２５は、図１の対応する名称の、ブロック構成部１、信号遅延部２、平面推定部３、定常性方向評価部４、および予測タップ構成部５と同様の機能を有するものであり、その説明は適宜省略する。
【００３６】
予測係数学習装置に入力されたＳＤ信号は、上述したように、ブロック構成部２１、信号遅延部２２、平面推定部２３、および定常性方向評価部２４によりクラス分類される。分類クラスに対して、予測係数学習部２６により、予測係数が算出される。以下、予測係数学習部２６により行われる予測係数の算出について説明する。
【００３７】
いま、注目画素の画素データｙの予測値Ｅ［ｙ］を、その注目画素と空間的または時間的に近接する位置にある画素（注目画素を含む）の入力データｘ１，ｘ２，ｘ３，・・・と、所定の予測係数ｗ１，ｗ２，ｗ３，・・・の線形結合により規定される線形１次結合モデルにより求める場合、予測値Ｅ［ｙ］は、次式で表すことができる。
Ｅ［ｙ］＝ｗ１ｘ１＋ｗ２ｘ２＋ｗ３ｘ３＋・・・・・・（４）
【００３８】
式（４）を一般化した例として、予測係数ｗの集合でなる行列Ｗ、入力データｘでなる行列Ｘ、および予測値Ｅ［ｙ］の集合でなる行列Ｙを、
【数２】

と定義すると、次式（５）のような観測方程式が成立する。
観測方程式：ＸＷ＝Ｙ ・・・（５）
【００３９】
そして、この観測方程式に最小自乗法を適用して注目画素の画素データｙに近い予測値Ｅ［ｙ］を求めることを考える。この場合、教師データとなる注目画素の真の画素データｙの集合でなる行列Ｙ’、および画素データｙに対する予測値Ｅ［ｙ］の残差ｅの集合でなる行列Ｅを、
【数３】

で定義すると、式（５）から次式のような残差方程式（６）が成立する。
残差方程式：ＸＷ＝Ｙ＋Ｅ ・・・（６）
【００４０】
なお、教師データとは、参照ＳＤ画像のことであり、入力ＳＤ信号と同一内容であるが、ノイズ成分のない非ノイズ画像である。
【００４１】
画素データｙに近い予測値Ｅ［ｙ］を求めるための予測係数ｗ_iは、自乗誤差
【数４】

を最小にすることで求めることができる。従って、この自乗誤差を予測係数ｗ_iで微分したものが０になる場合の予測係数ｗ_i、すなわち、次式（７）を満たす予測係数ｗ_iが、画素データｙに近い予測値Ｅ［ｙ］を求めるための最適値ということになる。
【数５】

【００４２】
そこで、まず、式（６）を微分することにより次式（８）が成立する。
【数６】

【００４３】
式（７）と式（８）より次式（９）が得られる。
【数７】

【００４４】
さらに、式（６）の残差方程式における学習データｘ、予測係数ｗ、教師データの画素データｙ、および残差ｅの関係を考慮すると、式（９）から、次のような正規方程式（１０）を得ることができる。
【００４５】
【数８】

【００４６】
式（１０）の正規方程式は、求めるべき予想係数ｗの数と同じ数だけたてることができるので、式（１０）を解くことで、最適な予測係数ｗを求めることができる。なお、式（１０）を解くにあたっては、例えば、掃き出し法（Gauss-Jordanの消去法）などを適用することが可能である。
【００４７】
このようにして求められた予測係数ｗは、クラス（予測タップ）と関連付けられて予測係数ROM６（図１）に記憶される。予測処理部７は、上述したようにして求めれれた予約係数ROM６に記憶されている予約係数ｗを用いて、式（４）に示した線形１次結合モデルにより、注目画素に対しての適応処理を行う。
【００４８】
このようにして算出された予測係数の一例を図８に示す。図８には、クラス１５２とクラス２０８８の予測係数を示している。このような予測係数ｗと予測タップに対応する画素データｙを、式（４）に代入することにより、ノイズが除去された画素データを得ることが可能となる。
【００４９】
本実施の形態においては、レベル値が等しいと判断された定常方向に関し、クラス分類適応処理を用いたので、動画にも静止画にも最適なノイズの除去が可能となる。
【００５０】
上述した説明においては、ノイズ除去に関して本発明を適用したが、動き検出にも適用することが可能である。すなわち、本発明は、クラス分類する際に、レベル値が等しいと判断される平面を推定し、さらに定常方向を判定する。このことは、例えば、定常方向が時間方向に関して、右上方向である場合、その画像の被写体は、右上の方向に移動していると判断することが可能であることを示している。
【００５１】
従って、分類クラスにより、被写体の動き方向を検出する事が可能である。本実施の形態を動き検出に用いた場合、ノイズがのった画像に対しても、ノイズののった状態のデータを用いて定常方向を判定できるので、ノイズの影響を受け難い動き検出が可能となる。
【００５２】
本明細書中において、上記処理を実行するコンピュータプログラムをユーザに提供する提供媒体には、磁気ディスク、CD-ROMなどの情報記録媒体の他、インターネット、デジタル衛星などのネットワークによる伝送媒体も含まれる。
【００５３】
【発明の効果】
以上の如く本発明によれば、動画にも静止画にも適したノイズ成分の除去を行うことが可能となる。
【図面の簡単な説明】
【図１】本発明を適用して画像処理装置の一実施の形態の構成を示すブロック図である。
【図２】 図１に示した画像処理装置の動作を説明するフローチャートである。
【図３】時空間ブロックを説明する図である。
【図４】推定平面を説明する図である。
【図５】選択候補画素を説明する図である。
【図６】クラスと予測タップの一例を示す図である。
【図７】予測係数学習装置の構成を示すブロック図である。
【図８】クラスと予測係数の一例を示す図である。
【符号の説明】
１ ブロック構成部， ２ 信号遅延部， ３ 平面推定部， ４ 定常性方向評価部， ５ 予測タップ構成部， ６ 予測係数ROM， ７ 予測処理部，２６ 予測係数学習部[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image processing apparatus and method, in particular, it relates to a suitable image processing apparatus and method used to remove the noise present in the time direction in the image data.
[0002]
[Prior art]
Conventionally, in order to remove noise components of image data, image motion detection is performed, and filtering in the time direction is performed on a portion (pixel group) that is determined to be sufficiently stationary. .
[0003]
[Problems to be solved by the invention]
However, the noise component removal process described above has a problem that correct motion detection cannot be performed when the noise components are superimposed, and as a result, correct noise components cannot be removed.
[0004]
In addition, the noise component removal process described above suppresses noise components for image areas that are determined to be sufficiently stationary, but attempts to suppress noise components for moving image areas as well. To perform noise removal processing by smoothing in a uniform space (regardless of structurally fixed or main image attribute) (to remove the changed part as noise), the spatial resolution of the main image There was a problem that it was accompanied by deterioration.
[0005]
The present invention has been made in view of such a situation, and in a predetermined number of fields, a plane that is determined to have the same signal level is estimated, and the estimated plane has an inclination of 0. One direction is determined, and adaptive processing is performed so that noise is removed in that direction, thereby suppressing the degradation of the spatial resolution.
[0006]
[Means for Solving the Problems]
An image processing apparatus according to the present invention includes a reference unit that includes a reference field in which a pixel of interest is located in the center, and blocks pixel data including a predetermined number of fields positioned before or after the reference field in time, By substituting the pixel data blocked by the blocking means into a predetermined expression, the plane estimation means for estimating the plane that is determined to have the same level value, and the plane expression estimated by the plane estimation means Substituting the coordinate values of a plurality of pixels set in, and calculating the dynamic range for each class by taking the difference between the minimum value and the maximum value, and storing the prediction coefficient set for each class Storage means for reading out, and reading means for reading out the prediction coefficient corresponding to the class having the minimum value of the dynamic range calculated by the calculation means from the storage means; And a adaptive processing means for performing adaptive processing using the prediction taps formed of the prediction coefficient and predetermined pixels read by the reading means, a predetermined equation of the plane estimating means, the coordinates of the pixel of interest (horizontal x , Vertical y, time z) = (0, 0, 0), the coordinates of other pixels in the block are expressed, the signal level is r, the residual is e, and the coefficients are c1 to c4.
r + e = c1 * x + c2 * y + c3 * z + c4
The plane estimation means estimates a plane that minimizes the residual e by the method of least squares .
[0007]
The image processing method of the present invention includes a reference step in which a pixel of interest is located in the center, and a blocking step for blocking pixel data composed of a predetermined number of fields located before or after the reference field, By substituting the pixel data blocked in the blocking step into a predetermined formula, the plane estimation step for estimating the plane that is determined to have the same level value, and the plane formula estimated in the plane estimation step Substituting the coordinate values of a plurality of pixels set in, and calculating the dynamic range for each class by taking the difference between the minimum and maximum values, and storing the prediction coefficient set for each class And the prediction coefficient corresponding to the class having the minimum value of the dynamic range calculated in the calculation step is read from the storage step. A reading step of issuing an adaptive processing step of performing adaptive processing using the prediction taps formed of the prediction coefficient and predetermined pixels read by the reading step, with a predetermined equation of the plane estimating step, the pixel of interest Coordinates are (horizontal x, vertical y, time z) = (0, 0, 0), the coordinates of other pixels in the block are expressed, signal level is r, residual is e, and coefficients are c1 to c4. When
r + e = c1 * x + c2 * y + c3 * z + c4
In the plane estimation step, the plane that minimizes the residual e is estimated by the method of least squares.
[0009]
In the image processing apparatus and method of the present invention , pixel data of a block composed of a predetermined number of fields is substituted into a predetermined formula, and a plane on which the level values are determined to be the same is estimated, and the estimated The coordinate value of multiple pixels set for each class is assigned to the plane formula of the plane, and the dynamic range for each class is calculated by taking the difference between the minimum and maximum values. The adaptive processing is performed using the prediction coefficient corresponding to the class to be held and the prediction tap made up of predetermined pixels. Further, the predetermined formula when the plane is estimated represents the coordinates of the other pixels in the block with the coordinates of the pixel of interest as (horizontal x, vertical y, time z) = (0, 0, 0), When the signal level is r, the residual is e, and the coefficients are c1 to c4,
r + e = c1 * x + c2 * y + c3 * z + c4
The plane that minimizes the residual e is estimated by the method of least squares.
[0010]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the present invention will be described below, but in order to clarify the correspondence between each means of the invention described in the claims and the following embodiments, in parentheses after each means, The features of the present invention will be described with the corresponding embodiment (however, an example) added. However, of course, this description does not mean that each means is limited to the description.
[0011]
The image processing apparatus according to claim 1, wherein the image processing apparatus includes a reference field in which a target pixel is located at the center, and blocks pixel data including a predetermined number of fields located before or after the reference field in time. A plane estimation unit (for example, the block configuration unit 1 in FIG. 1) and a plane estimation unit that estimates pixel planes determined to have the same level value by substituting the pixel data blocked by the blocking unit into a predetermined expression. For example, by substituting the coordinate values of a plurality of pixels set for each class in the plane equation estimated by the plane estimation means in step S3) of FIG. 2, and taking the difference between the minimum value and the maximum value Calculation means for calculating the dynamic range for each class (for example, step S4 in FIG. 2) and storage means for storing the prediction coefficient set for each class (for example, the prediction range in FIG. 1). Coefficient ROM 6), reading means (for example, step S5 in FIG. 2) for reading out the prediction coefficient corresponding to the class having the minimum value of the dynamic range calculated by the calculating means, and prediction read by the reading means An adaptive processing means (for example, step S6 in FIG. 2) that performs an adaptive process using a coefficient and a prediction tap made up of predetermined pixels is provided.
[0012]
FIG. 1 is a block diagram showing a configuration of an embodiment of an image processing apparatus to which the present invention is applied. An input SD (Standard Definition) signal is supplied to a signal delay unit 2 configured by a FIFO (First In First Out) or the like via a block configuration unit 1. The SD signal input to the signal delay unit 2 is read as a block by the block configuration unit 1 as necessary, and is output to the plane estimation unit 3. The signal output from the plane estimation unit 3 is output to the stationarity direction evaluation unit 4.
[0013]
The signal output from the stationarity direction evaluation unit 4 is output to a prediction tap configuration unit 5 and a prediction coefficient ROM (Random Access Memory) 6. The prediction processing unit 7 performs a predetermined process on the signal output from the prediction tap configuration unit 5 using the prediction coefficient stored in the prediction coefficient ROM 6, and outputs the processed SD signal.
[0014]
Next, the operation of the image processing apparatus will be described with reference to the flowchart of FIG. In step S1, an SD signal (for example, an 8-bit PCM (Pulse Code Modulation) luminance signal) is input to the signal delay unit 2 via the block configuration unit 1 and stored. In step S <b> 2, the block configuration unit 1 reads a predetermined amount of SD signal (image data) stored in the signal delay unit 2. As shown in FIG. 3A, the predetermined amount of data to be read is a total composed of five first fields (for example, odd fields) and four second fields (for example, even fields). Of the data for nine fields, the data is for a predetermined area of each field. For simplicity, in this example, as shown in FIG. 3B, the first field area is composed of 45 (= 9 × 5) pixels, and the second field area is 36 (= 9 × 4) pixels. It shall consist of. Therefore, the total number of pixels in the region for nine fields read from the signal delay unit 2 is 369 pixels. The 9 fields composed of 369 pixels are hereinafter referred to as local space-time blocks as appropriate.
[0015]
In other words, the local spatio-temporal block includes a reference field in which the target pixel to be processed is located in the center of the region, and is composed of a total of nine fields of four fields located before or after the reference field in time. The The reference field region is a field composed of 9 pixels in the horizontal direction and 5 pixels in the vertical direction. When the coordinate value in the time axis direction of the reference field is set to 0, the coordinate value (time) in the time axis direction of the field that exists in time before the reference field is expressed as minus and exists in time after the reference field. The coordinate value (time) in the time axis direction of the field to be displayed is represented by plus. Accordingly, the coordinate value (time) in the time axis direction changes from −4 to 4. The spatio-temporal coordinates of the pixel of interest are shown as follows, and the coordinates of other pixels are expressed using this coordinate as the origin.
(Horizontal, vertical, time) = (x, y, z) = (0, 0, 0)
[0016]
The block configuration unit 1 collectively reads data of one local spatiotemporal block from the signal delay unit 2 and outputs the data to the plane estimation unit 3. The plane estimation unit 3 substitutes all the pixel data of the input local spatiotemporal block into the following equation (1).
r _n + e = c 1 · x _n + c 2 · y _n + c 3 · z _n + c 4 (1)
In the formula (1), r _n is the luminance signal value of the pixel data of the space-time coordinates (horizontal, vertical, time) _{(x n, y n, z} n) in the noise image (input SD image), e is The residual, (x _n , y _n , z _n ) is a coordinate value with the target pixel of the horizontal, vertical, and time of the nth pixel in the spatiotemporal block as the origin, and c1 to c4 are coefficients.
[0017]
The plane estimation unit 3 substitutes the input luminance signal value r _n of the pixel data of the local spatio-temporal block and the coordinate values (x _n , y _n , z _n ) of the pixel data into the equation (1). The coefficients c1 to c4 are obtained so that the square sum of the residual e shown in the following equation (2) is minimized.
[Formula 1]

[0018]
Note that the value m in Expression (2) is 368 in the example of FIG. When the obtained coefficients are c1 ′ to c4 ′, a plane expression shown in Expression (3) is generated.
r = c1 ′ · x + c2 ′ · y + c3 ′ · z + c4 ′ (3)
[0019]
In this way, the plane generated by the equation (3) expressed using the coefficients c1 ′ to c4 ′ obtained by the plane estimation unit 3 is set as the estimation plane. An example of this estimated plane is shown in FIG. The estimation plane shown in FIG. 4 is a plane generated by the equation (3), and is a plane generated by pixels actually existing in the local spatiotemporal block, in other words, a local area on the estimation plane. It is not necessarily the plane on which the pixels of the space-time block are placed. In other words, the estimated plane is a plane that exists at a position where pixels that are determined to have substantially the same signal level (luminance value) will exist.
[0020]
When the estimation plane as shown in FIG. 4 is estimated, that is, in the horizontal direction of the surface composed of the vertical direction and the horizontal direction, the vertical coordinate value decreases from the left side to the right side, and in the time direction, the horizontal and vertical directions When a plane with no change in coordinate values is estimated, an image whose luminance value gradually decreases or rises (like gradation) from the upper right side to the lower left side in the figure during the time of 9 fields It will be displayed without change.
[0021]
Note that the estimation plane shown in FIG. 4 represents an isolevel plane, and does not represent a signal level itself, but is represented as a contour line of a signal level value. .
[0022]
As another example in which the estimation plane as shown in FIG. 4 is estimated, a direction perpendicular to the estimation plane is used, such as a case where the luminance value gradually changes with respect to the previous spatial direction or a steep change on the stairs. Is an image that exhibits some luminance value change.
[0023]
Next, the stationarity direction evaluation unit 4 is included in the estimated plane represented using the coefficients c1 ′ to c4 ′ obtained by the plane estimation unit 3 in this way, and the inclination is at least one-dimensional direction. The direction of 0 (steady direction) is obtained. The direction of the obtained inclination 0 is a linear expression expressed by a one-dimensional or two-dimensional function, for example, f (x) or f (x, y). In the case of an image that has a staircase shape when the above-described luminance value is plotted as it is, the obtained steady direction is the direction of the step of the staircase, in other words, the direction orthogonal to the direction of going up and down the staircase.
[0024]
The stationarity direction evaluation unit 4 obtains the stationarity direction and also performs class classification. In other words, the steady direction based on the class is determined. That is, a plurality of classes and conditions for classification into each class are determined in advance, and continuity is determined depending on which class is classified. The condition for classification into each class depends on the position of a predetermined 5 pixels among 369 pixels included in 9 fields.
[0025]
In this case, the total number of classes is ₃₆₉ C ₅ classes. However, the ₃₆₉ C ₅ class is not practical because of the huge number of combinations. Therefore, as shown in FIG. 5, when considering the number of classes using 5 pixels from predetermined 21 pixels (hereinafter, referred to as selection candidate pixels as appropriate) of the 5 first fields, ₂₁ C ₅ = 20349 The number of classes is easy to handle. Further, by adding a condition that the target pixel is necessarily included, four pixels are selected from 20 selection candidate pixels, so that ₂₀ C ₄ = 4845 classes, which is a more easily handled class number.
[0026]
In the example shown in FIG. 5, 17 selection candidate pixels in the reference field are arranged so that 5 pixels can be selected in all directions related to the vertical and horizontal diagonal directions including the target pixel. In the fields before and after the reference field, four points that differ only in the coordinate value of the target pixel from the time axis, in other words, four points that have the same vertical and horizontal coordinate values as the target pixel are selected candidate pixels. Yes.
[0027]
In the example of FIG. 5, only the pixels existing in the first field are used. However, the pixels existing in the second field may be used as selection candidate pixels, or pixels of 21 pixels or more are selected. It is also good. In other words, it is best to select 396 pixels, which are all the pixels in the nine fields, as the selection candidate pixels, but since it is not practical as described above, select as many pixels as possible with a practical number. It is desirable to use candidate pixels.
[0028]
If a class is created using 21 selection candidate pixels including the target pixel, 4845 classes can be created. In FIG. 6, among the 4845 classes, class 0 and class 4844 (first and last classes) and characteristic classes 152 and 2088 are shown as examples. Class numbering is based on the scan order. For example, the class 0 is a pixel having a coordinate value (0, 0, −4) that is scanned first, and a coordinate value (0, 0, −2) that is scanned next, among the 21 selection candidate pixels. The pixel, the pixel of the coordinate value (−4, −4, 0) to be scanned next, the pixel of the coordinate value (0, −4, 0) to be scanned next, and the coordinate value (0, (0, 0) is a total of 5 pixels.
[0029]
Similarly, classes 0 to 4844 are generated by assigning class numbers based on the scan order. The class 152 is a class in which images, such as still images, in which the stationary direction is most stable in the time axis direction are classified. The class 2088 is a class in which an image having the most continuity in the diagonal direction within the same field is classified, for example, an image in which a single line is diagonally drawn on a white background.
[0030]
The stationarity direction evaluation unit 4 first substitutes the coordinate values of five pixels in classes 0 to 4844 in order into the estimated plane equation (3) obtained by the plane estimation unit 3, and is calculated for each of the five pixels. The difference (dynamic range) between the minimum value and the maximum value of the value r. Then, among the dynamic ranges obtained for each class, the class having the smallest dynamic range is set as the classification class of the local spatiotemporal block being processed. However, the value r calculated by the continuity direction evaluation unit 4 in this way does not mean a luminance value, but is simply used as a value for calculating a dynamic range.
[0031]
To describe in detail, first, coordinate values of five pixels of class 0, that is, (0,0, -4), (0,0, -2), (-4, -4,0), (0, By substituting (−4, 0), (0, 0, 0) sequentially into equation (3), five values r are obtained. The dynamic range of class 0 is obtained by taking the difference between the minimum value and the maximum value of the obtained five values r. By performing the same processing for classes 1 to 4844, a total of 4845 dynamic ranges can be obtained. Of these 4845 dynamic ranges, the class having the minimum dynamic range is determined as the class of the pixel being processed (hereinafter referred to as a classification class). If a class with a dynamic range of 0 exists, it means that 5 pixels of that class exist on the estimated plane represented by Expression (3).
[0032]
In this way, the obtained classification class is supplied to the prediction tap configuration unit 5 and the prediction coefficient ROM 6. In step S <b> 5, the prediction tap configuration unit 5 reads out pixel data of the prediction tap corresponding to the supplied classification class from the signal delay unit 2 and outputs the pixel data to the prediction processing unit 7. Further, the prediction coefficient ROM 6 outputs a prediction coefficient corresponding to the supplied classification class to the prediction processing unit 7. As the prediction tap, as shown in FIG. 6, 5 pixels set for each class used to determine the steady direction are used. Therefore, the pixel data read from the signal delay unit 2 by the prediction tap configuration unit 5 is pixel data (luminance value) existing at a position corresponding to the coordinate value of the five pixels of the classification class.
[0033]
In step S6, the prediction processing unit 7 performs an adaptive process using the supplied prediction coefficient and pixel data, and outputs the processed pixel data. Note that the adaptive processing is processing for performing a linear first combination model calculation represented by Expression (4), which will be described later, using a prediction coefficient corresponding to a class of a pixel of interest and pixel data of a prediction tap.
[0034]
As described above, in the image processing apparatus, the input SD image is adaptively processed for each class using the prediction coefficient stored in the prediction coefficient ROM 6 in advance. Here, a prediction coefficient learning device that generates a prediction coefficient stored in the prediction coefficient ROM 6 will be described.
[0035]
FIG. 7 is a block diagram illustrating a configuration of the prediction coefficient learning apparatus. The block configuration unit 21, the signal delay unit 22, the plane estimation unit 23, the stationarity direction evaluation unit 24, and the prediction tap configuration unit 25 correspond to the block configuration unit 1, the signal delay unit 2, the plane of the corresponding names in FIG. It has functions similar to those of the estimation unit 3, the stationarity direction evaluation unit 4, and the prediction tap configuration unit 5, and description thereof will be omitted as appropriate.
[0036]
As described above, the SD signals input to the prediction coefficient learning device are classified into classes by the block configuration unit 21, the signal delay unit 22, the plane estimation unit 23, and the stationarity direction evaluation unit 24. A prediction coefficient is calculated by the prediction coefficient learning unit 26 for the classification class. Hereinafter, calculation of the prediction coefficient performed by the prediction coefficient learning unit 26 will be described.
[0037]
Now, the predicted value E [y] of the pixel data y of the pixel of interest is converted into the input data x1, x2, x3,... Of the pixels (including the pixel of interest) located spatially or temporally close to the pixel of interest. .., And a linear primary combination model defined by a linear combination of predetermined prediction coefficients w1, w2, w3,..., The prediction value E [y] can be expressed by the following equation.
E [y] = w1x1 + w2x2 + w3x3 + (4)
[0038]
As an example of generalizing Equation (4), a matrix W composed of a set of prediction coefficients w, a matrix X composed of input data x, and a matrix Y composed of a set of predicted values E [y]
[Expression 2]

Then, an observation equation such as the following equation (5) is established.
Observation equation: XW = Y (5)
[0039]
Then, it is considered to apply the least square method to this observation equation to obtain a predicted value E [y] close to the pixel data y of the target pixel. In this case, a matrix Y ′ composed of a set of true pixel data y of the target pixel serving as teacher data and a matrix E composed of a set of residuals e of predicted values E [y] for the pixel data y are
[Equation 3]

Is defined from equation (5), the following residual equation (6) is established.
Residual equation: XW = Y + E (6)
[0040]
The teacher data is a reference SD image, which is a non-noise image having the same content as the input SD signal but having no noise component.
[0041]
The prediction coefficient w _i for obtaining the predicted value E [y] close to the pixel data y is a square error ## EQU4 ##

Can be obtained by minimizing. Accordingly, the prediction coefficient w _{i when the} square error is differentiated by the prediction coefficient w _i , that is, the prediction coefficient w _i satisfying the following equation (7) is the predicted value E [y ] Is the optimum value for obtaining the above.
[Equation 5]

[0042]
Therefore, first, the following formula (8) is established by differentiating the formula (6).
[Formula 6]

[0043]
The following equation (9) is obtained from the equations (7) and (8).
[Expression 7]

[0044]
Further, considering the relationship among the learning data x, the prediction coefficient w, the pixel data y of the teacher data, and the residual e in the residual equation of Equation (6), from Equation (9), the following normal equation (10 ) Can be obtained.
[0045]
[Equation 8]

[0046]
Since the normal equation of Expression (10) can be formed by the same number as the number of prediction coefficients w to be obtained, the optimal prediction coefficient w can be obtained by solving Expression (10). In solving equation (10), for example, a sweep-out method (Gauss-Jordan elimination method) or the like can be applied.
[0047]
The prediction coefficient w thus obtained is stored in the prediction coefficient ROM 6 (FIG. 1) in association with the class (prediction tap). The prediction processing unit 7 uses the reservation coefficient w stored in the reservation coefficient ROM 6 obtained as described above, and adapts to the target pixel using the linear first combination model shown in Expression (4). Process.
[0048]
An example of the prediction coefficient calculated in this way is shown in FIG. FIG. 8 shows prediction coefficients of class 152 and class 2088. By substituting the prediction coefficient w and the pixel data y corresponding to the prediction tap into Equation (4), it is possible to obtain pixel data from which noise has been removed.
[0049]
In the present embodiment, since the class classification adaptive processing is used for the steady direction in which the level values are determined to be equal, it is possible to remove noise that is optimal for both moving images and still images.
[0050]
In the above description, the present invention is applied to noise removal, but it can also be applied to motion detection. That is, according to the present invention, when classifying, a plane that is determined to have the same level value is estimated, and a steady direction is further determined. This indicates that, for example, when the steady direction is the upper right direction with respect to the time direction, it is possible to determine that the subject of the image is moving in the upper right direction.
[0051]
Therefore, it is possible to detect the movement direction of the subject by the classification class. When this embodiment is used for motion detection, a steady direction can be determined using noise-added data even for a noise-added image, so that motion detection that is less susceptible to noise can be performed. It becomes possible.
[0052]
In this specification, the medium for providing a computer program for executing the above processing to the user includes not only an information recording medium such as a magnetic disk and a CD-ROM, but also a transmission medium via a network such as the Internet and a digital satellite. .
[0053]
【The invention's effect】
As described above , according to the present invention, it is possible to remove noise components suitable for both moving images and still images.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating a configuration of an embodiment of an image processing apparatus to which the present invention is applied.
FIG. 2 is a flowchart for explaining the operation of the image processing apparatus shown in FIG.
FIG. 3 is a diagram illustrating a spatiotemporal block.
FIG. 4 is a diagram illustrating an estimation plane.
FIG. 5 is a diagram illustrating selection candidate pixels.
FIG. 6 is a diagram illustrating an example of a class and a prediction tap.
FIG. 7 is a block diagram illustrating a configuration of a prediction coefficient learning device.
FIG. 8 is a diagram illustrating an example of a class and a prediction coefficient.
[Explanation of symbols]
1 block configuration unit, 2 signal delay unit, 3 plane estimation unit, 4 stationarity direction evaluation unit, 5 prediction tap configuration unit, 6 prediction coefficient ROM, 7 prediction processing unit, 26 prediction coefficient learning unit

Claims (4)

Blocking means for blocking pixel data including a reference field in which a pixel of interest is located in the center and composed of a predetermined number of fields positioned before or after the reference field in time,
Plane estimation means for substituting the pixel data blocked by the blocking means into a predetermined formula and estimating a plane determined to have the same level value;
A calculation that calculates the dynamic range for each class by substituting the coordinate values of a plurality of pixels set for each class into the plane equation estimated by the plane estimation means and taking the difference between the minimum value and the maximum value. Means,
Storage means for storing a prediction coefficient set for each class;
Reading means for reading out the prediction coefficient corresponding to the class having the minimum value of the dynamic range calculated by the calculating means from the storage means;
Adaptive processing means for performing adaptive processing using the prediction coefficient read by the reading means and a prediction tap comprising predetermined pixels ;
With
The predetermined formula of the plane estimation means is
The coordinates of the pixel of interest are (horizontal x, vertical y, time z) = (0, 0, 0), the coordinates of other pixels in the block are expressed, the signal level is r, the residual is e, and the coefficient is When c1 to c4,
r + e = c1 * x + c2 * y + c3 * z + c4
The plane estimation means estimates the plane that minimizes the residual e by the method of least squares . The image processing apparatus according to claim 1, wherein the adaptive processing unit calculates data from which noise of the pixel of interest has been removed by linear linear combination of the prediction tap and the prediction coefficient. The image processing apparatus according to claim 1, wherein a motion of an image displayed by the pixel data is detected from a class having a minimum value of a dynamic range calculated by the calculation unit. A blocking step for blocking pixel data including a reference field in which a pixel of interest is located at the center, and a predetermined number of fields positioned before or after the reference field in time;
A plane estimation step for substituting the pixel data blocked in the blocking step into a predetermined formula and estimating a plane determined to have the same level value;
A calculation that calculates the dynamic range for each class by substituting the coordinate values of a plurality of pixels set for each class into the plane equation estimated in the plane estimation step and taking the difference between the minimum value and the maximum value. Steps,
Storing a prediction coefficient set for each class;
A reading step of reading out the prediction coefficient corresponding to the class having the minimum value of the dynamic range calculated in the calculation step from the storage step;
An adaptive processing step for performing adaptive processing using the prediction coefficient read in the reading step and a prediction tap composed of predetermined pixels ;
With
The predetermined expression of the plane estimation step is
The coordinates of the pixel of interest are (horizontal x, vertical y, time z) = (0, 0, 0), the coordinates of other pixels in the block are expressed, the signal level is r, the residual is e, and the coefficient is When c1 to c4,
r + e = c1 * x + c2 * y + c3 * z + c4
In the image processing method , the process of the plane estimation step estimates a plane that minimizes the residual e by the method of least squares .

Images