JP3557626B2 - Image restoration apparatus and method - Google Patents

Description

【００３８】
なお、本来ＡＤＲＣはＶＴＲ（ビデオテープレコーダ）向けの高能率符号化用に開発された適応的再量子化法であり、信号レベルの局所的なパターンを短い語長で効率的に表現できるので、本実施例ではこれを信号パターンのクラス分類のコード発生に使用している。
【００３９】
より一般的に説明する。
学習信号画素レベルをｘ_１ ，・・・，ｘ_ｎ とし、原信号画素レベルをｙとしたとき、クラス毎に係数ｗ_１ ，・・・，ｗ_ｎ によるｎタップの線形推定式（２）
ｙ＝ｗ_１ ｘ_１ ＋ｗ_２ ｘ_２ ＋・・・＋ｗ_ｎ ｘ_ｎ （２）
を設定する。なお、学習前はｗ_ｉ が未定係数である。
【００４０】
学習は、クラス毎に複数の信号データに対して行う。データ数がｍの場合、式（２）（予測式） に従って、
ｙ_ｋ ＝ｗ_１ ｘ_ｋ１＋ｗ_２ ｘ_ｋ２＋・・・＋ｗ_ｎ ｘ_ｋｎ （ｋ＝１，２，・・・，ｍ） （３）
Ｍ＞ｎの場合にはｗ_１ ，・・・，ｗ_ｎ は一意には定まらないので、誤差ベクトルｅの要素を
ｅ_ｋ ＝ｙ_ｋ −｛ｗ_１ ｘ_ｋ１＋ｗ_２ ｘ_ｋ２＋・・・ ＋ｗ_ｎ ｘ_ｋｎ｝ （ｋ＝１，２，・・・，ｍ ）（４）
と定義して、式（５） を最小にする係数を求める。いわゆる最小自乗法による解法である。
【００４１】
【数２】

【００４２】
ここで、式（５） のｗ_ｉ による遍微分係数を、式（６） のように求める。
【００４３】
【数３】

【００４４】
上記式（６） を０にするように各ｗ_ｉ を決めればよいから、式（７） 、式（８） のように、
【００４５】
【数４】

【００４６】
【数５】

【００４７】
として行列を用いると式（９） のようになり、
【００４８】
【数６】

【００４９】
掃き出し法などの一般的な行列解法を用いて、ｗ_ｉ について解けば予測係数が求まり、本実施例ではＲＯＭのアドレスｃｌａｓｓのデータ（ＲＯＭテーブル）としてこの予測係数を格納しておく。
【００５０】
次に、以下に上記劣化関数について説明する。
先ず、雑音を考慮に入れた画像観測モデルを考える。この雑音を考慮に入れた画像観測モデルは式（１０）のように表すことができる。
ｇ＝Ａｆ＋ｎ （１０）
なお、この式（１０）において、ｆは原画像、ｇは観測データ、Ａは観測過程、ｎは加法性雑音を表している。
【００５１】
ここで、対象としている原画像ｆの属する空間をＨ_１ とし、観測データｇの属する空間をＨ_２ とすると、雑音ｎはＨ_２ の元であり、観測結果Ａは数学的にはＨ_１ やＨ_２ への作用素とみなすことができる。ｆやｇは、連続領域の上で定義された関数として表現されることもあるし、離散点上で定義された配列として表現されることもある。後者の場合、それはベクトルと同じものである。これらの表現方法に応じて、式（１０）のモデルは、次のように呼ばれる。すなわち、Ｈ_１ とＨ_２ が共に関数空間の場合は連続−連続モデルと言い、共にベクトル空間の場合は離散−離散モデルという。また、Ｈ_１ が関数空間でＨ_２ がベクトル空間の場合を、連続−離散モデルという。
【００５２】
上記式（１０）は、上記連続−連続モデルの場合、式（１１）のように表現される。
【００５３】
【数７】

【００５４】
なお、この式（１１）において、ｆ（ｘ，ｙ）は原画像、ｇ（ｘ，ｙ）は劣化画像、ｎ（ｘ，ｙ）は雑音であり、ｈ（ｘ，ｙ，ｘ′，ｙ′）は劣化関数と呼ばれるものである。例えば、ｆ（ｘ，ｙ）＝δ（ｘ−α，ｙ−β）なる点光源に対する劣化画像がｈ（ｘ，ｙ，α，β）である。したがって、劣化関数は同時に、点ひろがり関数（ＰＳＦ：ｐｏｉｎｔ ｓｐｒｅａｄ ｆｕｎｃｔｉｏｎ）と呼ばれることもある。
【００５５】
ここで、点光源の劣化した像が平行移動を除いて、その点の位置によらないとき、点ひろがり関数はｈ（ｘ−ｘ′，ｙ−ｙ′）と差形の関数になり、上記式（１１）は、式（１２）のように、畳み込み積分（ｃｏｎｖｏｌｕｔｉｏｎ） で表される。
【００５６】
【数８】

【００５７】
このような劣化を移動不変、位置不変、或いは空間不変であるという。変数変換によって、式（１１）を式（１２）の形式に変換できる場合もある。
【００５８】
また、劣化をｘ軸方向とｙ軸方向に分離できる場合がある。数学的に言えば、劣化関数が、式（１３）のように、積の形に分解できることに対応している。
ｈ（ｘ，ｙ，ｘ′，ｙ′）＝ｈ_１ （ｘ，ｘ′）ｈ_２ （ｙ，ｙ′） （１３）
これを分離核という。カメラの一様な動きによる写真のブレは、分離核かつ移動不変な劣化の例である。
【００５９】
また、上記式（１０）の離散−離散モデルは、次のようになる。例えば、原画像ｆ（ｘ，ｙ）のＮ個の標本点における値から作られるＮ次元ベクトルをｆとする。同様に劣化画像ｇ（ｘ，ｙ）及び雑音ｎ（ｘ，ｙ）か作られるＮ次元ベクトルを、それぞれｇ，ｎとすれば、上記式（１１）は近似的に、
ｇ＝Ａｆ＋ｎ （１４）
と表すことができる。ここで、式（１４）におけるＡは、劣化を表すＮ次正方行列である。
【００６０】
また、標本点がＮ_１ ×Ｎ_２ 個（Ｎ＝Ｎ_１ ×Ｎ_２ ）の格子点（ｘ_ｍ ，ｙ_ｎ ）で与えられている場合、ｆ（ｘ，ｙ）の標本値ｆ（ｘ_ｍ ，ｙ_ｎ ）をそのまま並べて、行列の形で表すことができる。これを〔ｆ〕とする。劣化画像ｇ（ｘ，ｙ）及び雑音ｎ（ｘ，ｙ）に対しても、同様に〔ｇ〕，〔ｎ〕とする。さらに、劣化Ａは式（１３）のように分離核になっているとする。このとき、連続−連続モデルの式（１１）は近似的に離散−離散モデルによって、
〔ｇ〕＝Ａ_１ 〔ｆ〕Ａ_２ ＋〔ｎ〕 （１５）
と表現できる。ここで、Ａ_１ ，Ａ_２ は、それぞれＮ_１ 次及びＮ_２ 次正方行列である。
【００６１】
さらに、劣化Ａが移動不変の場合、式（１２）に対して、式（１５）のＡ_１ ，Ａ_２ はテープリッツ（Ｔｏｅｐｌｉｔｚ） 行列で近似すれば、離散フーリエ変換（ＤＦＴ）によって対角行列に変換することができる。これは式（１２）の畳み込み積分をフーリエ変換によって積の形に変換することに相当している。
【００６２】
次に、連続−離散モデルは、連続−連続モデルによる劣化の後、標本化を行ってベクトルｇ或いは行列形式〔ｇ〕に変換したものである。したがって、前半の部分には式（１１）， 式（１５）の表現を使うことができる。或いは、この２つの過程をまとめて、式（１６）のように表現することもできる。
【００６３】
【数９】

【００６４】
この式（１６）で、ｇ_ｉｊ及びｎ_ｉｊは、それぞれ〔ｇ〕及び〔ｎ〕の第ｉｊ成分である。
【００６５】
なお、上述の式（１１）〜式（１９）に示したように、画像劣化モデルを具体的に表現する方法は数多くある。
【００６６】
図１に戻って、上述のようにして学習して得たＲＯＭテーブルからデータを読みだして画像復元を行う第１の実施例の画像復元装置の構成例について説明する。図１の画像復元装置は、入力劣化画像に対して劣化を復元した画素を出力する場合の構成を示している。
【００６７】
この図１において、入力端子１に入力された劣化したディジタル画像信号は、ブロック化回路２に送られる。当該ブロック化回路２では、供給されたディジタル画像信号を複数の画素データからなる複数のブロックに分割する。
【００６８】
上記ブロック化回路２からのブロックデータは、クラスコード発生回路３に送られる。当該クラスコード発生回路３では、供給されたブロックの２次元的なレベル分布のパターンを参照してクラスを求め、そのクラスコードを発生する。
【００６９】
このクラスコードは、アドレスデータとしてＲＯＭテーブル４に送られる。当該ＲＯＭテーブル４には、上述したようにした学習値（すなわち劣化無しの画素データ）が格納されており、上記クラスコード（アドレス）に対応してその学習値が出力される。当該ＲＯＭテーブル４から読みだされた学習値すなわち劣化無しの画素データが出力端子５から出力される。
【００７０】
次に、本発明の第２の実施例として、図５には、ＲＯＭテーブル９にブロック単位の学習値を保存している場合の構成を示す。
【００７１】
この図５において、入力端子６からクラスコード発生回路８までは、図１の入力端子１からクラスコード発生回路３までと同様の構成である。ここで、図１のＲＯＭテーブル４がブロック単位でないデータを出力するのに対し、図５のＲＯＭテーブル９はブロック単位のデータを出力する。
【００７２】
当該ＲＯＭテーブル９から出力される劣化無しのブロック単位のデータは、ブロック分解回路１０に送られる。すなわち、ＲＯＭテーブル９の出力は、ブロック単位であるため、ブロック分解回路１０によってブロック分解を行い、その分解されたデータが出力端子１１から劣化無し画像の出力データとなる。この第２の実施例では、ＲＯＭテーブル９とブロック分解回路１０とで無劣化画像出力手段が構成されている。
【００７３】
次に、図６には、本発明の第３の実施例として、劣化関数の逆関数で劣化を復元する復元フィルタの出力と図１，図５同様のブロック化回路１３の出力との差分について、先に学習を行っておいた場合についての例を示す。
【００７４】
すなわち、この図６に示す第３の実施例の画像復元装置のクラスコード発生回路１６では、ブロック化回路１３を介してブロック化された入力画像と、そのブロック画像を復元フィルタ１４を介してフィルタリングした出力との差分に対してクラス分類処理を施すようにしている。
【００７５】
この図６において、入力端子１２には図１，図５の入力端子１，６と同様にディジタル画像信号が供給され、同じく図１，図５と同様のブロック化回路１３でブロック化がなさる。
【００７６】
当該ブロック化回路１３の出力は、劣化関数の逆関数で画像の劣化を復元する復元フィルタ１４に送られる。当該フィルタ１４でフィルタリングがなされた出力は、加算器１５に減算信号として送られる。このとき当該加算器１５には上記ブロック化回路１３からの出力が加算信号としても供給され、したがって、当該加算器１５ではフィルタリングされたブロックデータとフィルタリングされていないブロックデータとの差分が求められることになる。
【００７７】
この加算器１５の出力は、クラスコード発生回路１６に送られる。当該クラスコード発生回路１６でも上記加算器１５の出力ブロックの２次元的なレベル分布のパターンを参照してクラスを求め、そのクラスコードを発生する。
【００７８】
このクラスコードは、アドレスデータとしてＲＯＭテーブル１７に送られる。当該ＲＯＭテーブル１７には、劣化の無い画素データと当該劣化の無い画素データを劣化フィルタでフィルタリングしたデータとの差分のデータで学習を行った学習値が格納されており、したがって、上記クラスコード（アドレス）に対応して学習値が出力されるようになる。
【００７９】
このＲＯＭテーブル１７から出力された学習値は、上記フィルタ１４の出力が加算信号として供給されている加算器１８に、同じく加算信号として送られる。これにより、当該加算器１８からは、劣化無しの画素データが出力され、これが出力端子１９から出力される。したがって、この第３の実施例装置では、上記ＲＯＭテーブル１７と、フィルタ１４の出力が供給される加算器１８とで無劣化画像出力手段が構成されている。
【００８０】
なお、第３の実施例の画像復元装置では、前記差分に対してクラス分類を施すようにしているが、例えば入力画像のフィルタ１４の出力に対してクラス分類を行うことも可能である。
【００８１】
上述のように、本発明実施例の画像復元においては、画質が劣化した入力画像を走査変換（ブロック化）し、このブロック画像データからクラス生成を行い、クラス（すなわち信号パターン）に応じて予め学習されたテーブルから画像復元に作用するデータ（学習最適化されたデータ）を出力し、そのデータから復元画像を生成することにより、劣化が修復された画像を得ることができ、したがって、劣化の原因が特定できない場合や、逆関数が一様に決定できない場合であっても、劣化した画像の復元が可能となる。
【００８２】
すなわち、画質が劣化した画像を復元しようとする場合には、通常、劣化原因を解析し、劣化モデル関数を決定し、逆関数を施すことにより復元を行うが、本発明実施例では、劣化の原因が特定できない場合や逆関数が一様に決定できない場合のために、劣化モデル関数の逆関数を用いる代わりに、学習により予め得られた復元テーブルを用いて、劣化した画像の代わりに劣化の無い復元画像を当てはめる（マッピング）することで、画像を復元するようにしている。
【００８３】
このように、本発明実施例では、劣化のない画像を用いた学習テーブルによるマッピングで画像を復元するため、複雑な劣化関数の決定や、その逆関数を求める必要がない上に、一度テーブルを作成すれば、処理は画像復元のテーブルを読み出すのみであるため、処理が簡単である。
また、劣化関数の逆関数が定義できず、従来なら復元できない場合においても学習により復元テーブルを作成することができる。
【００８４】
したがって、本実施例の画像復元装置によれば、例えばテレビカメラによって撮像された画像に対する画質劣化が修復された画像を得ることができる。
【００８５】
【発明の効果】
上述したように、本発明においては、画質が劣化した画像をブロック毎にクラス分類したデータと劣化の無い画像のブロックデータとで予め学習した結果を記憶しておき、劣化画像を複数のブロックに分割し、ブロック化されたデータを複数のクラスに分類してその分類に対応するクラスコードを得て、このクラスコードに応じて劣化画像に対応する劣化の無い画像を出力するようにしているため、劣化の原因が特定できない場合や、逆関数が一様に決定できない場合であっても、劣化した画像の復元が可能となる。
【図面の簡単な説明】
【図１】本発明の第１の実施例の画像復元装置の概略構成を示すブロック回路図である。
【図２】本発明実施例における学習についての第１の具体例のフローチャートである。
【図３】本発明実施例における学習についての第２の具体例のフローチャートである。
【図４】学習信号画像から原信号画像の予測を説明するための図である。
【図５】第２の実施例の画像復元装置の概略構成を示すブロック回路図である。
【図６】第３の実施例の画像復元装置の概略構成を示すブロック回路図である。
【符号の説明】
２，７，１３・・・・ブロック化回路
３，８，１６・・・・クラスコード発生回路
４，９，１７・・・・ＲＯＭテーブル
１０・・・・・・・・ブロック分解回路
１４・・・・・・・・フィルタ[0001]
[Industrial applications]
The present invention relates to an image restoration apparatus and method for restoring (restoring) a deteriorated image to an image without deterioration.
[0002]
[Prior art]
Conventionally, when restoring an image with deteriorated image quality, the cause of the deterioration is analyzed to obtain a model function of the deterioration, and an inverse function corresponding to the model function of the deterioration is applied to the deteriorated image. To restore the image.
[0003]
[Problems to be solved by the invention]
By the way, the cause of the deterioration may be caused by a uniform movement of a camera (an image photographing device such as a video camera) or various things such as blurring caused by an optical system of the camera. Therefore, when restoring the image, the model function of the deterioration differs depending on the cause of the deterioration.
[0004]
For this reason, when trying to restore the deterioration, if the cause of the deterioration cannot be identified, the model function cannot be determined, and the restoration becomes difficult.
[0005]
Further, even if a model function for the cause of deterioration can be established, there are many cases where there is no inverse function for restoration, and it is difficult to evaluate an optimal model in such a case.
[0006]
In other words, when restoring an image with deteriorated image quality, the cause of deterioration is analyzed, a deterioration model function is determined, and restoration is performed by applying its inverse function, but when the cause of deterioration cannot be specified as described above. If the inverse function cannot be determined uniformly, restoration becomes difficult.
[0007]
In view of the above, the present invention provides an image restoration apparatus and method capable of restoring a deteriorated image even when the cause of deterioration cannot be specified or the inverse function cannot be determined uniformly. It is intended to provide.
[0008]
[Means for Solving the Problems]
The present invention has been proposed to achieve the above-described object, and an image restoration apparatus according to the present invention includes a blocking unit that divides an input image that has deteriorated into a plurality of blocks, Classifying means for classifying the divided data into a plurality of classes based on the level distribution pattern of the data and outputting a class code corresponding to the classification, and an image having no deterioration corresponding to the input image according to the class code And a non-deteriorated image output means for outputting the image data.
Also, the image restoration apparatus according to the present invention divides the degraded input image into a plurality of blocks, and outputs the blocked data by a blocking unit. A classifying unit that classifies the blocked data into a plurality of classes and outputs a class code corresponding to the classification, and that there is no deterioration corresponding to the input image according to the class code. Non-degraded image output means for outputting an image, wherein the class classification means is configured to generate a plurality of the blocked data based on a difference value between the output data from the restoration unit and the blocked data. Are characterized by the following classification.
[0009]
Here, the non-deteriorated image output means outputs data obtained by blocking an image having no deterioration and data obtained by classifying an image obtained by degrading the blocked image having no deterioration by a predetermined deterioration function. The result (learning value) learned in advance is stored as a table corresponding to the class code. Alternatively, the non-deteriorated image output means associates a result (learning value) obtained by learning in advance with data obtained by blocking the degraded image into classes and data obtained by blocking non-degraded images in correspondence with the class code. It is stored as a table.
[0010]
An image restoration method according to the present invention includes a blocking step of dividing a degraded input image into a plurality of blocks, and classifying the blocked data into a plurality of classes based on a level distribution pattern of the data. A class classification step of outputting a class code corresponding to the classification, and a non-deteriorated image output step of outputting a non-degraded image corresponding to the input image according to the class code.
Also, the image restoration method according to the present invention includes a blocking step of dividing an input image in which deterioration has occurred into a plurality of blocks and outputting blocked data, and a step of deteriorating the input image by an inverse function of the deterioration function. A classifying step of classifying the block-divided data into a plurality of classes and outputting a class code corresponding to the classification, and no deterioration corresponding to the input image according to the class code. A non-deteriorated image output step of outputting an image, wherein the class classification step includes the step of dividing the blocked data into a plurality based on a difference value between the output data from the restoration step and the blocked data. Are characterized by the following classification.
[0011]
Here, in the non-deteriorated image output step, data obtained by blocking an image having no deterioration and data obtained by classifying an image obtained by deteriorating the blocked image having no deterioration by a predetermined deterioration function are used. A learning value corresponding to the class code is extracted from a plurality of learning values obtained by learning in advance. Alternatively, in the non-degraded image output step, the class code corresponding to the class code is obtained from a plurality of learning values obtained in advance by learning the data obtained by classifying the degraded image into blocks and classifying the image having no deterioration. Take out the learned value.
[0012]
[Action]
According to the present invention, a result (restoration table) obtained by learning in advance with data obtained by classifying an image having deteriorated image quality for each block and block data of an image having no deterioration is stored. That is, since the learning table learns between the deteriorated image and the non-degraded image, the learning value corresponding to the class code obtained by the class classification of the deteriorated image is extracted from this table (instead of the deteriorated image). By applying a restored image without deterioration (mapping), an image whose deterioration has been restored can be obtained. Further, once the table is created, the image is restored only by reading the table, so that the processing is simplified.
[0013]
【Example】
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
[0014]
The image restoration apparatus according to the first embodiment of the present invention is an apparatus for restoring an image in which the image quality of a digital image signal has deteriorated. As shown in FIG. A block circuit 2 for dividing an input image into a plurality of blocks (scan conversion of the input image), and a class classification for classifying the blocked data into a plurality of classes and outputting a class code corresponding to the classification A class code generating circuit 3 as means, and a ROM table 4 as non-deteriorated image output means for outputting an image without deterioration corresponding to the input image in accordance with the class code (outputting data which has been previously learned and optimized). It has the following.
[0015]
Here, the ROM table 4 is composed of data obtained by blocking an image without deterioration and data obtained by classifying an image obtained by degrading the blocked image without deterioration by a predetermined deterioration function described later. The learning result (learning value) is stored in advance as a table corresponding to the class code (first specific example described later). Alternatively, a result (learning value) obtained by learning in advance the data obtained by classifying the deteriorated image into blocks and the data obtained by blocking the image having no deterioration is stored as a table corresponding to the class code (described later). Second specific example). Further, the class code generating circuit 3 obtains a coefficient group of an expression representing a shape closest to the input image using a higher-order polynomial as described later as a model expression, and performs class classification according to the coefficient group.
[0016]
Here, prior to the description of the configuration of the image restoration apparatus according to the embodiment of the present invention, learning in image restoration and generation of a ROM table will be described.
[0017]
When trying to perform image restoration using a table that has been learned in advance, it is necessary to perform a learning process between a deteriorated image and a clean image without deterioration.
As a method of obtaining a deteriorated image used for learning, for example, there is a method in which a subject is fixed and a camera is moved to generate a flow image, or a blurred image is generated by intentionally shifting the focus of the camera.
[0018]
Here, FIG. 2 shows a first specific example of a processing flow when learning is performed using a deteriorated image.
[0019]
In FIG. 2, in step S9, an image without deterioration is supplied. In the next step S10, the image without deterioration is divided into blocks and sent to step S13.
Further, the image without deterioration in step S10 is deteriorated in step S11 using a known deterioration function. Then, in step S12, clustering for classifying the block data degraded by the above degradation function into classes is performed, and the process proceeds to step S13. It should be noted that the class classification is a pattern classification using the image data that is divided into blocks.
[0020]
In step S13, training (learning) is performed between the block data from step S10 without deterioration and the block data from step S12 degraded and classified.
[0021]
In step S14, it is determined whether or not the learning has been repeatedly performed on all the blocks constituting the image to the extent that all the class learning is sufficiently performed. If the determination in step S14 is NO, the process returns to step S10, and if the determination is YES, the process proceeds to step S15.
[0022]
In step S15, a mapping table is generated with the optimal values learned for each class. This is the ROM table.
[0023]
In the first specific example, the degraded image is generated by deteriorating the non-degraded image by the degraded function. However, as in the second specific example described below, the camera movement and the focus Learning can also be performed using a deteriorated image generated by shifting or the like, and the processing flow in this case is as shown in FIG. 3, for example.
[0024]
In FIG. 3, in step S1, a deteriorated image physically generated in advance is supplied, and in step S2, an image without deterioration is supplied. In step S3, the degraded image is blocked, and in step S4, the non-degraded image is blocked at a position corresponding to the blocking in step S3.
[0025]
In step S5 following step S3, clustering for classifying the block data of the deteriorated image into classes is performed. This class classification is also a pattern classification using the image data that is divided into blocks.
[0026]
In step S6, which proceeds after the processing in steps S5 and S4, training (learning) is performed between the blocks of the degraded image and the non-degraded image.
[0027]
In step S7, it is determined whether or not the learning has been repeatedly performed on all the blocks constituting the image to the extent that all the class learning is sufficiently performed. If the determination in step S7 is NO, the process returns to step S3 for a deteriorated image, and returns to step S4 for an image without deterioration. If the determination is yes in step S7, the process proceeds to step S8.
[0028]
In step S8, a mapping table is generated with the optimal values learned for each class. This is the ROM table.
[0029]
Note that there are several patterns depending on how the blocks of the non-degraded image corresponding to the blocks of the degraded image are learned. For example, there is a case where learning is performed on a block of a deteriorated image and a block of a non-degraded image and output, or a case where learning is performed on pixels and output. Further, there is a case where the input image itself is learned as data to be learned, and a case where a difference amount from the deterioration filter is learned.
[0030]
Although real data can be used for the classes in the first and second specific examples, data compression or the like may be used for class classification in consideration of a memory capacity considering feasibility. .
[0031]
Examples of the data compression include adaptive quantization (referred to as ADRC) according to a dynamic range of an image signal described later, DPCM (differential quantization), vector quantization (VQ), and the like.
[0032]
Note that the ADRC means that a digital image signal is divided into blocks, a dynamic range defined by a maximum value and a minimum value of a plurality of pixel data included in the block is determined, and the dynamic range is adapted to the dynamic range determined in block units. Each pixel data in the block is encoded with the allocated number of bits.
More specifically, the ADRC is a dynamic range (difference between the maximum level and the minimum level in a block) defined by the maximum value and the minimum value of a plurality of pixels included in a predetermined block in one frame (or field). ) Is determined, the variable number of quantization bits adapted to the dynamic range is determined for each block, and the input data from which the minimum value has been removed is coded (requantized, that is, the compressed quantum The dynamic range is evenly divided according to the number of coding bits, and each pixel in the block is coded into a code of the closest level) to obtain a coded code. Is an additional code, and these encoded codes and additional codes are obtained.
[0033]
Further, the following method is used as a method for implementing clustering.
[0034]
Here, as the simplest method of class division, for example, there is a method of directly using a bit sequence of learning data in a block as a class number. However, this method requires a huge amount of ROM.
[0035]
For this reason, in the present embodiment, the above-described ADRC is used for class division based on a signal pattern, and the number of effective classes that preserves the properties of the signal pattern is reduced.
Here, for example, as shown in FIG. 4, taking the prediction from the learning signal pixels a, b, c to the original signal pixel A as an example, the levels of the learning signal images a, b, c are respectively x ₁ , X ₂ , X ₃ And The requantized data obtained by performing p-bit ADRC on the data of x1, x2, and x3 is q ₁ , Q ₂ , Q ₃ And the dynamic range is DR and the minimum value is min.
[0036]
At this time, the class number (class code) class of this block is defined by Expression (1).
[0037]
(Equation 1)

[0038]
Note that ADRC is an adaptive requantization method originally developed for high-efficiency coding for a VTR (video tape recorder), and can efficiently represent a local pattern of a signal level with a short word length. In this embodiment, this is used to generate a code for classifying a signal pattern.
[0039]
This will be described more generally.
Let the learning signal pixel level be x ₁ , ..., x _n And when the original signal pixel level is y, the coefficient w ₁ , ..., w _n N-tap linear estimation formula (2)
y = w ₁ x ₁ + W ₂ x ₂ + ... + w _n x _n (2)
Set. Before learning, w _i Is an undetermined coefficient.
[0040]
The learning is performed on a plurality of signal data for each class. When the number of data is m, according to equation (2) (prediction equation),
y _k = W ₁ x _k1 + W ₂ x _k2 + ... + w _n x _kn (K = 1, 2,..., M) (3)
W if M> n ₁ , ..., w _n Is not uniquely determined, the element of the error vector e is
e _k = Y _k − ｛W ₁ x _k1 + W ₂ x _k2 + ... + w _n x _kn ｝ (K = 1, 2,..., M) (4)
And find the coefficient that minimizes equation (5). This is a so-called least squares method.
[0041]
(Equation 2)

[0042]
Here, w in Expression (5) _i Is obtained as in equation (6).
[0043]
(Equation 3)

[0044]
Each w is set so that the above equation (6) becomes zero. _i Since equation (7) and equation (8) can be determined,
[0045]
(Equation 4)

[0046]
(Equation 5)

[0047]
When a matrix is used as Equation (9),
[0048]
(Equation 6)

[0049]
Using a general matrix solution such as a sweeping method, w _i Is solved, a prediction coefficient is obtained. In this embodiment, the prediction coefficient is stored as data (ROM table) of the address class of the ROM.
[0050]
Next, the deterioration function will be described below.
First, consider an image observation model that takes noise into account. An image observation model taking this noise into account can be expressed as in equation (10).
g = Af + n (10)
In the equation (10), f represents an original image, g represents observation data, A represents an observation process, and n represents additive noise.
[0051]
Here, the space to which the target original image f belongs is H ₁ And the space to which the observation data g belongs is H ₂ Then, the noise n is H ₂ Observation result A is mathematically H ₁ And H ₂ Operator. f and g may be expressed as a function defined on a continuous area, or may be expressed as an array defined on discrete points. In the latter case, it is the same as the vector. Depending on these representations, the model of equation (10) is called as follows: That is, H ₁ And H ₂ When both are function spaces, they are called continuous-continuous models, and when both are vector spaces, they are called discrete-discrete models. Also, H ₁ Is H in the function space ₂ Is a continuous space model.
[0052]
Equation (10) is expressed as equation (11) in the case of the continuous-continuous model.
[0053]
(Equation 7)

[0054]
In equation (11), f (x, y) is an original image, g (x, y) is a degraded image, n (x, y) is noise, and h (x, y, x ′, y). ') Is what is called a degradation function. For example, a deteriorated image for a point light source where f (x, y) = δ (x−α, y−β) is h (x, y, α, β). Therefore, the degradation function may also be referred to as a point spread function (PSF) at the same time.
[0055]
Here, when the deteriorated image of the point light source does not depend on the position of the point except for the parallel movement, the point spread function becomes a function of h (xx ′, yy ′) and a difference form. Equation (11) is expressed by convolution as shown in equation (12).
[0056]
(Equation 8)

[0057]
Such deterioration is referred to as movement-invariant, position-invariant, or space-invariant. In some cases, equation (11) can be converted to the form of equation (12) by variable conversion.
[0058]
In some cases, deterioration can be separated in the x-axis direction and the y-axis direction. Mathematically speaking, this corresponds to the fact that the degradation function can be decomposed into a product as shown in Expression (13).
h (x, y, x ', y') = h ₁ (X, x ') h ₂ (Y, y ') (13)
This is called a separation nucleus. Photo blurring due to uniform movement of the camera is an example of separation nucleus and movement invariant degradation.
[0059]
The discrete-discrete model of the above equation (10) is as follows. For example, an n-dimensional vector created from values at N sample points of the original image f (x, y) is defined as f. Similarly, assuming that the N-dimensional vectors generated from the degraded image g (x, y) and the noise n (x, y) are g and n, the above equation (11) is approximately
g = Af + n (14)
It can be expressed as. Here, A in equation (14) is an N-th order square matrix representing deterioration.
[0060]
Also, if the sampling point is N ₁ × N ₂ (N = N ₁ × N ₂ ) Grid point (x _m , Y _n ), The sample value f (x, y) of f (x, y) _m , Y _n ) Can be expressed as a matrix by arranging them as they are. This is [f]. [G] and [n] are similarly set for the deteriorated image g (x, y) and the noise n (x, y). Further, it is assumed that the deterioration A is a separation nucleus as shown in the equation (13). At this time, equation (11) of the continuous-continuous model is approximately expressed by a discrete-discrete model.
[G] = A ₁ [F] A ₂ + [N] (15)
Can be expressed as Where A ₁ , A ₂ Is N ₁ Next and N ₂ It is a square matrix.
[0061]
Further, when the deterioration A is invariant in movement, the expression (15) ₁ , A ₂ If is approximated by a Toeplitz matrix, it can be converted to a diagonal matrix by a discrete Fourier transform (DFT). This corresponds to transforming the convolution integral of Expression (12) into a product form by Fourier transform.
[0062]
Next, the continuous-discrete model is obtained by performing sampling and converting into a vector g or a matrix form [g] after deterioration by the continuous-continuous model. Therefore, the expressions of Expressions (11) and (15) can be used in the first half. Alternatively, these two processes can be collectively expressed as Expression (16).
[0063]
(Equation 9)

[0064]
In this equation (16), g _ij And n _ij Is the ij-th component of [g] and [n], respectively.
[0065]
Note that, as shown in the above equations (11) to (19), there are many methods for specifically expressing the image deterioration model.
[0066]
Returning to FIG. 1, a description will be given of a configuration example of the image restoration apparatus according to the first embodiment for reading data from the ROM table obtained by learning as described above and restoring an image. The image restoration device in FIG. 1 shows a configuration in which pixels obtained by restoring the deterioration of an input deteriorated image are output.
[0067]
In FIG. 1, a deteriorated digital image signal input to an input terminal 1 is sent to a blocking circuit 2. The blocking circuit 2 divides the supplied digital image signal into a plurality of blocks including a plurality of pixel data.
[0068]
The block data from the blocking circuit 2 is sent to the class code generating circuit 3. The class code generation circuit 3 obtains a class by referring to the two-dimensional level distribution pattern of the supplied block, and generates the class code.
[0069]
This class code is sent to the ROM table 4 as address data. The ROM table 4 stores the learning values described above (that is, pixel data without deterioration), and outputs the learning values corresponding to the class codes (addresses). The learning value read from the ROM table 4, that is, the pixel data without deterioration is output from the output terminal 5.
[0070]
Next, as a second embodiment of the present invention, FIG. 5 shows a configuration in a case where learning values in block units are stored in the ROM table 9.
[0071]
In FIG. 5, the configuration from the input terminal 6 to the class code generation circuit 8 is the same as the configuration from the input terminal 1 to the class code generation circuit 3 in FIG. Here, the ROM table 4 in FIG. 1 outputs data that is not in block units, whereas the ROM table 9 in FIG. 5 outputs data in block units.
[0072]
The data in the block unit without deterioration output from the ROM table 9 is sent to the block decomposition circuit 10. That is, since the output of the ROM table 9 is in units of blocks, block decomposition is performed by the block decomposition circuit 10, and the decomposed data becomes output data of a non-deteriorated image from the output terminal 11. In the second embodiment, the ROM table 9 and the block decomposition circuit 10 constitute a non-deteriorated image output unit.
[0073]
Next, FIG. 6 shows, as a third embodiment of the present invention, the difference between the output of the restoration filter for restoring the deterioration by the inverse function of the deterioration function and the output of the blocking circuit 13 similar to FIGS. An example of a case where learning has been performed first will be described.
[0074]
That is, in the class code generation circuit 16 of the image restoration apparatus according to the third embodiment shown in FIG. 6, the input image blocked by the blocking circuit 13 and the block image are filtered by the restoration filter 14. Classification processing is performed on the difference from the output.
[0075]
In FIG. 6, a digital image signal is supplied to the input terminal 12 in the same manner as the input terminals 1 and 6 in FIGS. 1 and 5, and the input terminal 12 is blocked by the blocking circuit 13 similar to FIGS.
[0076]
The output of the blocking circuit 13 is sent to a restoration filter 14 for restoring the image degradation with the inverse function of the degradation function. The output filtered by the filter 14 is sent to the adder 15 as a subtraction signal. At this time, the output from the blocking circuit 13 is also supplied to the adder 15 as an addition signal. Therefore, the difference between the filtered block data and the unfiltered block data is obtained by the adder 15. become.
[0077]
The output of the adder 15 is sent to a class code generation circuit 16. The class code generation circuit 16 also obtains a class by referring to the two-dimensional level distribution pattern of the output block of the adder 15 and generates the class code.
[0078]
This class code is sent to the ROM table 17 as address data. The ROM table 17 stores learning values obtained by learning with difference data between pixel data having no deterioration and data obtained by filtering the pixel data having no deterioration by the deterioration filter. The learning value is output corresponding to the address.
[0079]
The learning value output from the ROM table 17 is also sent as an addition signal to an adder 18 to which the output of the filter 14 is supplied as an addition signal. As a result, the adder 18 outputs pixel data without deterioration, and this is output from the output terminal 19. Therefore, in the device of the third embodiment, the ROM table 17 and the adder 18 to which the output of the filter 14 is supplied constitute a non-degraded image output unit.
[0080]
In the image restoration apparatus according to the third embodiment, the classification is performed on the difference. However, for example, the classification may be performed on the output of the filter 14 of the input image.
[0081]
As described above, in the image restoration according to the embodiment of the present invention, an input image whose image quality has deteriorated is scan-converted (blocked), a class is generated from the block image data, and a class is generated in advance according to the class (ie, signal pattern). By outputting data (learning-optimized data) acting on image restoration from the learned table and generating a restored image from the data, it is possible to obtain an image whose deterioration has been repaired. Even if the cause cannot be specified or the inverse function cannot be determined uniformly, it is possible to restore a deteriorated image.
[0082]
That is, when an image with deteriorated image quality is to be restored, usually, the cause of the degradation is analyzed, a degradation model function is determined, and restoration is performed by applying an inverse function. In cases where the cause cannot be specified or the inverse function cannot be determined uniformly, instead of using the inverse function of the degradation model function, a restoration table obtained in advance by learning is used to replace An image is restored by applying (mapping) a restored image that does not exist.
[0083]
As described above, in the embodiment of the present invention, since an image is restored by mapping using a learning table using an image having no deterioration, it is not necessary to determine a complicated deterioration function or find an inverse function thereof. If created, the process is simple because the process only reads the table for image restoration.
In addition, even when the inverse function of the deterioration function cannot be defined and restoration cannot be conventionally performed, a restoration table can be created by learning.
[0084]
Therefore, according to the image restoration apparatus of the present embodiment, it is possible to obtain an image in which the image quality of an image captured by, for example, a television camera has been restored.
[0085]
【The invention's effect】
As described above, in the present invention, a result of learning in advance with data obtained by classifying an image with deteriorated image quality for each block and block data of an image having no deterioration is stored, and the deteriorated image is stored in a plurality of blocks. Because the divided and blocked data is classified into a plurality of classes, a class code corresponding to the classification is obtained, and an image without deterioration corresponding to the deteriorated image is output according to the class code. Even if the cause of the deterioration cannot be specified or the inverse function cannot be determined uniformly, the deteriorated image can be restored.
[Brief description of the drawings]
FIG. 1 is a block circuit diagram illustrating a schematic configuration of an image restoration device according to a first embodiment of the present invention.
FIG. 2 is a flowchart of a first specific example of learning in the embodiment of the present invention.
FIG. 3 is a flowchart of a second specific example of learning in the embodiment of the present invention.
FIG. 4 is a diagram for explaining prediction of an original signal image from a learning signal image.
FIG. 5 is a block circuit diagram illustrating a schematic configuration of an image restoration device according to a second embodiment.
FIG. 6 is a block circuit diagram illustrating a schematic configuration of an image restoration device according to a third embodiment.
[Explanation of symbols]
2,7,13... Blocking circuit
3, 8, 16,... Class code generation circuit
4, 9, 17, ... ROM table
10 ......... Block disassembly circuit
14 Filter

Claims (8)

Blocking means for dividing the degraded input image into a plurality of blocks;
Class classification means for classifying the blocked data into a plurality of classes based on a level distribution pattern of the data and outputting a class code corresponding to the classification;
An image restoration apparatus comprising: a non-deteriorated image output unit that outputs an image without deterioration corresponding to an input image according to the class code. The non-deteriorated image output means is configured to previously classify between data obtained by blocking a non-degraded image and data obtained by classifying an image obtained by degrading the blocked non-degraded image by a predetermined deterioration function. 2. The image restoration apparatus according to claim 1, wherein a result of learning is stored in association with the class code. The non-deteriorated image output means stores, in association with the class code, a result of learning in advance between data obtained by blocking and classifying a deteriorated image and data obtained by blocking an image having no deterioration. 2. The image restoration device according to claim 1, wherein: Blocking means for dividing the degraded input image into a plurality of blocks and outputting the blocked data,
A restoration unit for restoring the deterioration of the input image with an inverse function of the deterioration function ,
Class classification means for classifying the blocked data into a plurality of classes and outputting a class code corresponding to the classification;
Non-degraded image output means for outputting an image without deterioration corresponding to the input image according to the class code,
Said classification means, based on the difference value between the output data and the blocked data from the recovery unit, an image restoration apparatus characterized by classifying the blocked data into a plurality of classes. A blocking step of dividing the degraded input image into a plurality of blocks;
A class classification step of classifying the blocked data into a plurality of classes based on a level distribution pattern of the data and outputting a class code corresponding to the classification;
A non-deteriorated image output step of outputting an image without deterioration corresponding to the input image according to the class code. Above the lossless image output step, advance between the image without the data and the block of which block the image without deterioration deterioration subjected to classification on the image was degraded by a predetermined degradation function data 6. The image restoration method according to claim 5, wherein a learning value corresponding to the class code is extracted from a plurality of learning values obtained by learning. In the non-deteriorated image output step, a plurality of learning values obtained by learning in advance between data obtained by classifying a degraded image into blocks and data obtained by blocking non-degraded images in correspondence with the class code are used. 6. The image restoration method according to claim 5, wherein the learned value is extracted. A blocking step of dividing the degraded input image into a plurality of blocks and outputting the blocked data;
A restoration step of restoring the deterioration of the input image with an inverse function of the deterioration function ,
A class classification step of classifying the blocked data into a plurality of classes and outputting a class code corresponding to the classification;
A non-degraded image output step of outputting an image without deterioration corresponding to the input image according to the class code,
The image restoration method, wherein the class classification step classifies the blocked data into a plurality of classes based on a difference value between output data from the restoration step and the blocked data.

Images