JP3759537B2 - Image signal transmission apparatus and image signal transmission method - Google Patents

Description

でなる観測方程式の形で表わすことができる。但し、（１）式における、ｎは１つの補間対象画素ｙ_m を線形一次結合式によつて表わす際の空間的及び時間的に周辺の伝送画素数を表し（実施例の場合、１つの補間対象画素ｙ_m を３８タツプの線形一次結合モデルによつて表わすため、ｎ＝３８である）、ｍは１フレーム内に存在する補間対象画素数を表わす。
【００４９】
ここで（１）式に基づき、１フレームに対して１個の係数組を求めようとすると、（１）式から１フレームの補間対象画素数分の連立方程式を作ることになる。基本的には、この連立方程式を解いて係数を求めればよい。実施例では、最小二乗法を用いてこの連立方程式を解く。
すなわち先ず、（１）式を残差行列Ｅを用いて次式、
【数２】

のように残差方程式の形に表現し直す。
【００５０】
ここで（２）式から各係数値ｗ_i の最確値を求めるためには、ｅ₁ ²＋ｅ₂ ²＋……＋ｅ_m-1 ²＋ｅ_m ² を最小にする条件、すなわち次式、
【数３】

となるｎ個の条件を入れてこれを満足するｗ₁ 、ｗ₂ 、……、ｗ_n を見つければよい。
ここで（２）式より、次式、
【数４】

を得、（３）式の条件をｉ＝１、２、……、ｎについて立てればそれぞれ、次式
【数５】

が得られる。ここで（２）式及び（５）式から次式の正規方程式が得られる。
【数６】

【００５１】
ここで（６）式で表わされる正規方程式は未知数の数がｎ個の連立方程式であるから、これにより最確値である各係数ｗ_i を求めることができる。正確には（６）式でｗ_i に掛かる（Σｘ_jnｘ_jn）（但し、ｊ＝１……ｍ）のマトリクスが正則であれば解くことができる。実際には、Gauss-Jordanの消去法（掃き出し法）を用いて連立方程式を解く。
【００５２】
実施例の場合には、クラス分類回路４７によつて求めた各クラス毎に上述した最小二乗法を用いて係数値ｗ_i を求める。この結果各クラス毎に１フレームにつき１組の係数を伝送すればよいことになり、全ての補間対象画素についての係数値ｗ_i を求めて伝送する場合に比して、格段に伝送情報量及び演算量を低減し得る。実際に、係数の情報量はフレーム当りの画素情報量に比べて無視できるくらいのオーダーである。
【００５３】
具体的には、係数選定回路４８は、図８に示すように構成されている。すなわち係数選定回路４８はローカルデコードされた復号データＤ２１（ｘ_i ）及び入力画像データＤ１に含まれる補間対象画素データｙ_m を時系列変換メモリ９０に入力する。時系列変換メモリ９０は、図３について上述したように線形一次結合モデルを形成するための画素（ｘ₁ 〜ｘ_n 、ｙ_m ）を同時化して出力する。
時系列変換メモリ９０から出力されたデータは、正規化方程式生成回路９１に与えられ、当該正規化方程式生成回路９１によつて各クラス毎に（６）式で表わされるような正規化方程式が生成され、続くＣＰＵ演算回路９２によつて掃き出し法によつて各クラス毎の係数組が求められる。
【００５４】
正規化方程式生成回路９１は、先ず乗算器アレイ９３によつて各画素同士の乗算を行う。乗算器アレイ９３は、図９に示すように構成されており、四角で表わす各セル毎に画素同士の乗算を行い、これにより得た各乗算結果を続く加算器メモリ９４に与える。
【００５５】
加算器メモリ９４は、図１０に示すように、乗算器アレイ９３と同様に配列されたセルでなる加算器アレイ９５とメモリ（またはレジスタ）アレイ９６Ａ〜９６Ｇとにより構成されている。メモリアレイ９６Ａ〜９６Ｇはクラス数分（実施例の場合、静止平坦部を表わすクラス７を除く７個のクラス分）だけ設けられており、インデツクスデータＤ２２をデコードするインデツクスデコーダ９７の出力（クラス）に応答して一つのメモリアレイ９６Ａ〜９６Ｆ又は９６Ｇが選択され、選択されたメモリアレイ９６Ａ〜９６Ｆ又は９６Ｇの格納値が加算器アレイ９５に帰還される。このとき加算器アレイ９５によつて得られる加算結果が、再び対応するメモリアレイ９６Ａ〜９６Ｆ又は９６Ｇに格納される。
【００５６】
このようにして乗算器アレイ９３、加算器アレイ９５及びメモリアレイ９６Ａ〜９６Ｇによつて積和演算が行われ、インデツクスによつて決定されるクラス毎にメモリアレイ９６Ａ〜９６Ｆ又は９６Ｇが選択されて、積和演算の結果によつてメモリアレイ９６Ａ〜９６Ｇの内容が更新される。
【００５７】
なお、各々のアレイの位置は、（６）式で表わされる正規化方程式のｗ_i にかかる（Σｘ_jnｘ_jn）（但し、ｊ＝１……ｍ）の位置に対応する。（６）式の正規化方程式を見れば明らかなように右上の項を反転すれば左下と同じものになるため、各アレイは三角形の形状をしている。
【００５８】
このようにして、１フレームの間にクラス毎に積和演算が行われて各クラス毎の正規化方程式が生成される。クラス毎の正規化方程式の各項の結果は、それぞれのクラスに対応するメモリアレイ９６Ａ〜９６Ｇに記憶されており、次にそれらのクラス毎の正規化方程式の各項が掃き出し法演算を実現するＣＰＵ演算回路９２によつて計算される。この結果各クラス毎の係数ｗ_i の組が求められ、これが係数データＤ４として送出される。
【００５９】
補間データ推定回路５０及び６３は、図１１に示すように、１フレーム毎に各クラスの係数組を記憶する係数メモリ１００を有し、この係数メモリ１００は各クラス毎の係数組ｗ₁ 〜ｗ_n を格納すると共に、インデツクスデコーダ１０１の出力（クラス）に応答してクラスに応じた係数組ｗ₁ 〜ｗ_n を出力する。この係数組ｗ₁ 〜ｗ_n がそれぞれレジスタ１０２Ａ₁ 、１０２Ａ₂ 、……、１０２Ａ_n を介して乗算器１０３Ａ₁ 、１０３Ａ₂ 、……、１０３Ａ_n に与えられる。また乗算器１０３Ａ₁ 〜１０３Ａ_n には、時系列変換回路１０４により選択された復号データｘ₁ 〜ｘ_n が与えられる。従つて乗算器１０３Ａ₁ 〜１０３Ａ_n の出力が加算回路１０５により加算されることにより、補間対象画素の推定値ｙ（＝ｘ₁ ｗ₁ ＋ｘ₂ ｗ₂ ＋……＋ｘ_n ｗ_n ）が得られる。
【００６０】
（３）実施例の動作
以上の構成において、画像符号化装置４０は入力画像データＤ１に対して間引き処理を施すことによりデータ量を削減した後、圧縮符号化データＤ２を生成する。また画像符号化装置４０は、間引かれた画素を注目画素とし、当該注目画素の周辺の伝送画素の状態に応じて各注目画素をクラス分類し、次にこのクラス毎の係数組を求め、これを係数データＤ４として出力する。
【００６１】
さらに画像符号化装置４０は、誤差データ形成部４２において、画像復号化装置６０側で圧縮符号化データＤ２、係数データＤ４に基づいて得られるであろう推定補間データＤ２３を予め算出すると共に、この推定補間データＤ２３の真値からの誤差値Ｄ２４を算出し、当該誤差値Ｄ２４に基づく誤差データＤ２０を出力する。
【００６２】
ここで画像符号化装置４０の係数選定回路４８では、一種の統計的処理により係数データＤ４を生成するため、この係数データＤ４に基づいて補間値を生成すると必然的に補間値と真の画素値との間にはある程度の誤差が生じることになる。このためこの誤差のある補間値を用いて復元画像を生成すると、この誤差分だけ画質が劣化することになる。
これを回避するため、画像符号化装置４０では、圧縮符号化データＤ２及び係数データＤ４に加えて、誤差データＤ２０を伝送する。
【００６３】
画像復号化装置６０では、圧縮符号化データＤ２を復号した復号データＤ３０に基づき、画像符号化装置４０でしたのと同様のクラス分類処理を施し、これにより得たインデツクスデータＤ３１に応じて選択した係数データＤ４と復号化データＤ３０とを線形一次結合させることにより、推定補間データＤ３２を求める。
【００６４】
次に、画像復号化装置６０は、推定補間データＤ３２に、伝送された誤差コードＤ２０を逆量子化したものを加算することにより、推定補間データＤ３２に含まれる誤差分をキヤンセルする。
かくして入力画像データＤ１に含まれる実際の値にほぼ等しい値の補間データＤ３３を得ることができることにより画質の向上した復元画像を得ることができる。
【００６５】
（４）実施例の効果
以上の構成によれば、画像符号化装置４０において、クラス分類処理を行い、画像データ（圧縮符号化データＤ２）と共に各クラス毎の係数データＤ４を伝送し、画像復号化装置６０において伝送された画像データ（圧縮符号化データＤ２）と係数データＤ４に基づき補間画素を生成する場合に、画像符号化装置４０において予め推定補間画素値（推定補間データＤ２３）と真値（入力画像データＤ１）との差分値（誤差データＤ２０）を求め、当該差分値（誤差データＤ２０）を、画像データ（圧縮符号化データＤ２）及び係数データＤ４と共に伝送するようにしたことにより、画像復号化装置６０において画質劣化の少ない復元画像（復元画像データＤ３４）を得ることができる。
またこれによりクラス数を増やしたり、係数選定の際の演算量を増加させることなく復元画像（復元画像データＤ３４）の画質を向上させることができる。
【００６６】
（５）他の実施例
なお上述の実施例においては、圧縮エンコーダ４４としてＡＤＲＣ回路を用いた場合について述べたが、本発明はこれに限らず、圧縮エンコーダ４４として例えばＤＣＴ（Discrete Cosine Transform ）変換符号化、ＤＰＣＭ、ベクトル量子化、サブバンド符号化、ウエーブレツト変換等圧縮手法を用いた場合にも、上述の実施例と同様の効果を得ることができる。
【００６７】
また上述の実施例においては、間引き後に残つた伝送画素を圧縮エンコーダ４４により圧縮符号化して伝送する場合について述べたが、本発明はこれに限らず、圧縮符号化せずに間引き後に残つた間引きデータをそのまま伝送する場合にも適用することができる。この場合には、画像符号化装置４０側で圧縮エンコーダ４４及びローカルデコーダ４６を省略すると共に、画像復号化装置６０側で圧縮デコーダ６１を省略するようにすればよい。
【００６８】
また上述の実施例においては、補間対象画素（注目画素）ｙ_m の周辺の伝送画素Ｘ１〜Ｘ１４間で差分絶対値を求め、当該差分絶対値が最も小さい方向をその補間対象画素のクラスとするようなクラス分類を用いた場合について述べたが、本発明はこれに限らず、要は補間対象画素の周辺の伝送画素の状態に応じて各補間対象画素をクラス分類するようにすればよく、種々のクラス分類処理を適用することができ、またクラス数も種々の個数を用いることができる。
【００６９】
さらにクラス分類に用いる伝送画素を一旦ＡＤＲＣ符号化した後、クラス分類処理を行うようにすれば、クラス情報数を削減することができる。またこのようにＡＤＲＣ回路の出力に基づいてクラス分類を行うようにすれば、画面全体の画素レベルが変動したような場合にも、画像の特徴のみに基づいた正確なクラス分類を行うことができる。
【００７０】
また上述の実施例においては、クラス分類回路４７を設け、補間対象画素をその周辺の伝送画素の状態に応じてクラス分類し、係数選定回路４８において各クラス毎の係数組を求めるようにした場合について述べたが、本発明はクラス毎の係数組を求める場合に限らず、例えば画像の局所的な特徴に応じて画像を空間内で細分化し、細分化した領域毎に係数組を求めるようにした場合にも適用でき、この場合クラス分類回路４７及び６２を省略するようにすれば良い。これにより例えば図１２に示すような構成の画像符号化装置１及び画像復号化装置１０にも本発明を適用することができる。
【００７１】
また上述の実施例においては、注目画素とこの注目画素の空間的及び又は時間的に周辺の伝送画素との相関関係を表すパラメータとして、線形一次結合モデルを最小二乗法により解いて得た係数データＤ４を伝送する場合について述べたが、本発明はこれに限らず、パラメータとしては例えば各クラス毎に求めた注目画素の周辺の伝送画素値の平均値を代表値として伝送するようにしても良い。この場合平均演算に重心法を用いるようにすれば、クラス毎に誤差の少ない代表値を容易に求めることができる。またこれに限らず、要は伝送されない注目画素と伝送画素との相関関係を表すような種々のパラメータを用いることができる。
【００７２】
さらに上述の実施例においては、サブサンプリング回路４３によつて入力画像データＤ１に対して１／２の間引き処理を行う場合について述べたが、本発明はこれに限らず、例えば１／４の間引き処理等他の間引き処理を行つた場合でも上述の実施例と同様の効果を得ることができる。
【００７３】
【発明の効果】
上述のように本発明によれば、入力画像データの所定画素をオフセツトサンプリングによつて間引くことにより画素数の低減した伝送画素データを生成するサンプリング手段と、サンプリング手段により間引かれた画素を注目画素とし、当該注目画素を中心として空間的及び又は時間的に配列された複数の伝送画素の階調又はパターンに基づいて各注目画素をクラス分類する第１のクラス分類手段と、第１のクラス分類手段により分類されたクラスごとに、注目画素と当該注目画素を中心として空間的及び又は時間的に配列された複数の伝送画素とから線形一次結合モデルを立て、最小二乗法の演算により当該線形一次結合モデルを解いて得られる係数をパラメータとして、メモリ手段に一時記憶させるパラメータ算出手段と、クラスに応じたパラメータと注目画素を中心として空間的及び又は時間的に配列された複数の伝送画素とに基づく線形一次結合式を用いることにより注目画素の第１の推定画素値を算出する第１の推定画素値算出手段と、第１の推定画素値と入力画像データに含まれる真の画素値との差分値を算出する差分値算出手段とを設け、差分値を伝送画素データ及びパラメータと共に伝送するようにしたことにより、送信側で間引かれた画素を受信側で正確に補間でき、この結果復元画像の解像度を一段と向上し得る。
また上述のように本発明によれば、入力画像データの所定画素をオフセツトサンプリングによつて間引くことにより画素数の低減した伝送画素データを生成する第１のステツプと、第１のステツプにおいて間引いた画素を注目画素とし、当該注目画素を中心として空間的及び又は時間的に配列された複数の伝送画素の階調又はパターンに基づいて各注目画素をクラス分類する第２のステツプと、第２のステツプにおいて分類したクラスごとに、注目画素と当該注目画素を中心として空間的及び又は時間的に配列された複数の伝送画素とから線形一次結合モデルを立て、最小二乗法の演算により当該線形一次結合モデルを解いて得られる係数をパラメータとして、メモリ手段に一時記憶させる第３のステツプと、クラスに応じたパラメータと注目画素を中心として空間的及び又は時間的に配列された複数の伝送画素とに基づく線形一次結合式を用いることにより注目画素の第１の推定画素値を算出する第４のステツプと、第１の推定画素値と入力画像データに含まれる真の画素値との差分値を算出する第５のステツプと、差分値を伝送画素データ及びパラメータと共に伝送する第６のステツプとを設けるようにしたことにより、送信側で間引かれた画素を受信側で正確に補間でき、この結果復元画像の解像度を一段と向上し得る。
【図面の簡単な説明】
【図１】本発明を適用した画像符号化装置の全体構成を示すブロツク図である。
【図２】本発明を適用した画像復号化装置の全体構成を示すブロツク図である。
【図３】サブサンプリング回路による間引き処理の説明、並びに線形一次係合モデルに用いる画素及び係数の説明に供する略線図である。
【図４】圧縮エンコーダに用いたＡＤＲＣ回路、並びに圧縮デコーダ及びローカルデコーダに用いたＡＤＲＣ復号化回路の説明に供するブロツク図である。
【図５】クラス分類回路の構成を示すブロツク図である。
【図６】クラス分類の説明に供する略線図である。
【図７】コード化回路の構成を示すブロツク図である。
【図８】係数選定回路の構成を示すブロツク図である。
【図９】乗算器アレイの構成を示す略線図である。
【図１０】加算器メモリの構成を示す略線図である。
【図１１】補間データ推定回路の構成を示すブロツク図である。
【図１２】従来のデイジタルデータ変換装置の構成を示すブロツク図である。
【図１３】従来のデイジタルデータ変換装置の構成を示すブロツク図である。
【符号の説明】
４０……画像符号化装置、４１……符号化データ形成部、４２……誤差データ形成部、４３……サブサンプリング回路、４４……圧縮エンコーダ、４６……ローカルデコーダ、４７、６２……クラス分類回路、４８……係数選定回路、５０、６３……補間データ推定回路、５３……差分回路、６０……画像復号化装置、６４……加算回路、Ｄ１……入力画像データ、Ｄ２……圧縮符号化データ、Ｄ４……係数データ、Ｄ２０……誤差データ、Ｄ２２、Ｄ３１……インデツクスデータ、Ｄ２３、Ｄ３２……推定補間データ、Ｄ２４……誤差値、Ｄ３３……補間データ、Ｄ３４……復元画像データ、Ｗ₁ 〜Ｗ₃₈……係数。[0001]
【table of contents】
The present invention will be described in the following order.
Industrial application fields
Conventional technology (FIGS. 12 and 13)
Problems to be Solved by the Invention (FIGS. 12 and 13)
Means for Solving the Problems (FIGS. 1 to 3 and FIG. 6)
Action (FIGS. 1 to 3 and FIG. 6)
Example
(1) Overall configuration (FIGS. 1 and 2)
(2) Detailed configuration (FIGS. 3 to 11)
(3) Operation of the embodiment
(4) Effects of the embodiment
(5) Other embodiments
The invention's effect
[0002]
[Industrial application fields]
The present invention relates to an image signal transmission apparatus and an image signal transmission method, and is particularly suitable for application to an image signal transmission apparatus and an image signal transmission method for reducing the amount of information of original image data by thinning processing. .
[0003]
[Prior art]
Conventionally, in a so-called image signal transmission system that transmits an image signal to a remote place, such as a video conference system, or in an apparatus that digitizes an image signal and records and reproduces it on a video tape recorder or a video disk recorder, In order to efficiently use recording media, the amount of transmitted information and recorded information is reduced by efficiently encoding significant information using the correlation of digitized image signals, and the transmission efficiency and recording efficiency are increased. Has been made.
[0004]
Specifically, a technique for reducing the amount of data to be transmitted by performing high-efficiency compression coding on image data is widely used. In this high-efficiency encoding, in general, the original image data is transmitted by reducing the number of pixels by performing sub-sampling (decimation) in space or space-time in advance, and then performing compression encoding processing. The amount of information has been further reduced. An example of spatiotemporal subsampling is the MUSE (Multiple Sampling Encode) method.
[0005]
On the other hand, on the receiving side, the resolution is increased by obtaining by interpolation the pixels that are not transmitted, that is, the thinned pixels. In general, interpolation is performed using a fixed tap and a fixed coefficient filter.
However, in such an interpolation method, the interpolated pixel value is obtained by averaging the pixel values of a plurality of peripheral pixels around the interpolated pixel. Therefore, depending on the type of image, the image quality is blurred and the variation in time and space (that is, the jerkiness) , Edge business, etc.) occur, and as a result, there is a problem in that the restored image is deteriorated in image quality.
[0006]
As one method for solving this problem, a digital data conversion device as described in Japanese Patent Application No. 5-201913 has been proposed. As shown in FIG. 12A, the transmitter 1 in this digital data conversion apparatus generates compressed encoded data D2 by passing the input image data D1 through the sub-sampling circuit 2 and the encoder 3, and outputs it to the output terminal 4 as shown in FIG. To give.
[0007]
The transmitter 1 decodes the compressed encoded data D2 by the local decoder 5 and then supplies the compressed encoded data D2 to the least squares operation circuit 6. The least squares method arithmetic circuit 6 receives the decoded data D3 and the input image data D1, and uses the pixels thinned out by sub-sampling (removed pixels) as the target pixel, and the target pixel included in the input image data D1. A linear linear combination model is established with the pixel values and the pixel values of the surrounding pixels included in the decoded data D3, and the coefficients of the linear linear combination model are obtained by performing an operation of the least square method. As a result, the least square method arithmetic circuit 6 outputs coefficient data D4 corresponding to the pixels thinned out by the sub-sampling circuit 2, and the coefficient data D4 is given to the output terminal 7.
[0008]
Thus, in the transmitter 1, together with the compression encoded data D2 obtained by compressing and encoding the image data after sub-sampling, coefficient data D4 representing the correlation between the thinned pixels and the surrounding transmission pixels is obtained. It is designed to transmit.
[0009]
The receiver 10 of the digital data converter is configured as shown in FIG. 12B, and inputs the compression encoded data D2 to the decoder 12 through the input terminal 11. The decoded data D5 decoded by the decoder 12 is supplied to the time series conversion circuit 13 and the interpolation calculation circuit 14. The receiver 10 inputs the coefficient data D4 to the interpolation calculation circuit 14 via the input terminal 15.
[0010]
The interpolation calculation circuit 14 uses the decoded data D5 and the coefficient data D4 to obtain the interpolation data D6 from the linear linear combination formula, and sends this to the time series conversion circuit 13. The time series conversion circuit 13 arranges the decoded data D5 and the interpolation data D6 in the same manner as the original image (that is, the input image data D1), and obtains restored image data D7.
[0011]
Thus, in the digital data conversion apparatus constituted by the transmitter 1 and the receiver 10, the transmitter 1 side does not transmit only the compression-encoded data D2 obtained by high-efficiency encoding, but is thinned out. Coefficient data D4 representing the correlation between the pixel and its surrounding transmission pixels is transmitted together, and on the receiver 10 side, thinned pixels that are not actually transmitted are generated using the coefficient data D4. Thus, even when the amount of transmission information is reduced by thinning, image quality deterioration on the receiver 10 side can be reduced.
[0012]
As another method, as disclosed in Japanese Patent Laid-Open No. 5-328185, a transmission that has been subjected to thinning processing is provided on the receiving side in which a memory for storing a prediction coefficient or a prediction value obtained in advance by learning is provided. By generating an interpolated pixel value based on a prediction coefficient read from the memory according to image data, or by using a predicted value read from the memory according to transmission image data as an interpolated pixel value, image quality degradation based on thinning is reduced. Digital data conversion devices that reduce the number have been proposed.
[0013]
As shown in FIG. 13, in the digital data conversion apparatus, in the transmitter 20, the compressed encoded data D10 obtained through the sub-sampling circuit 21, the encoder 22 and the transmission processing circuit 23 is transferred to the transmission line 25 via the output terminal 24. Send it out.
The receiver 30 obtains the decoded data D11 by passing the compressed encoded data D10 inputted through the input terminal 31 through the reception processing circuit 32 and the decoder 33, and sends it to the synchronization circuit 34. The synchronization circuit 34 uses the thinned pixels as the target pixel, and synchronizes a plurality of peripheral transmission pixels for each target pixel and sends the same to the clustering circuit 35.
[0014]
The clustering circuit 35 classifies the input transmission pixel data according to the gradation and pattern, and sends the class classification result to the memory 36 as class information D12 representing the class of each pixel of interest.
The memory 36 outputs the prediction coefficient D13 corresponding to the input class information D12 among the prediction coefficients previously obtained and stored for each class by learning using the class information D12 as an address.
[0015]
The interpolation data creation circuit 37 creates the interpolation pixel data D14 by performing an operation by linear linear combination using the prediction coefficient D13 and the peripheral pixel value synchronized. The interpolated pixel data D14 and the decoded data D11 are combined by a subsequent combining circuit 38, whereby restored image data D15 is generated.
[0016]
Thus, according to the receiver 30, it is possible to restore high-frequency components that do not exist in the transmission pixel data, and as a result, it is possible to suppress deterioration in image quality based on the thinning process, and to obtain a high-resolution restored image. .
[0017]
[Problems to be solved by the invention]
By the way, in the method proposed in Japanese Patent Application No. 5-201913, the least square method arithmetic circuit 6 performs interpolation on a pixel to be interpolated (target pixel) and its surroundings spatially and temporally with respect to the pixel to be interpolated. A linear linear combination model with pixels is established, and coefficient data D4 is calculated from the linear linear combination model by the least square method.
[0018]
However, since this is a kind of statistical processing, the coefficient data D4 having a certain degree of error is necessarily calculated with respect to the actual pixel values included in the input image data D1. Therefore, on the receiver 10 side, there is a problem that an interpolated pixel value having an error with respect to the actual pixel value is generated by the error of this coefficient, and the image quality deteriorates.
Here, it is considered that the error in the coefficient data D4 becomes smaller as the number of peripheral pixels used in the linear linear combination model is increased. However, in this case, the calculation amount when calculating the coefficient data D4 increases, so that There is a problem that is difficult to realize.
[0019]
Further, as disclosed in Japanese Patent Laid-Open No. 5-328185, in the method of storing the prediction coefficient or the prediction value of the interpolation pixel obtained by learning in advance in the memory 36 for each class, as the number of classes is increased, In addition, it is considered that the interpolation pixel value closer to the true value can be generated as the number of peripheral pixels used for learning is increased. There is a problem that the amount of data that must be stored in the memory increases, resulting in an increase in size of the configuration. Moreover, even if the number of classes and the number of pixels used for learning were increased, the error with respect to the true value of the interpolated pixel value could not be completely eliminated.
[0020]
The present invention has been made in consideration of the above points, and when the input image data is subjected to thinning processing to reduce the amount of transmission data, the thinned pixels on the transmitting side are accurately interpolated on the receiving side. Thus, an image signal transmission apparatus and an image signal transmission method that can improve the resolution of a restored image are proposed.
[0021]
[Means for Solving the Problems]
  In order to solve such a problem, in the present invention, predetermined pixels of input image data are detected.OffsetSampling means for generating transmission pixel data in which the number of pixels is reduced by thinning out by sampling, and a pixel thinned out by the sampling means is a target pixel, and is arranged spatially and / or temporally around the target pixel. Multiple transmission pixelsGradation or patternA first class classifying unit that classifies each target pixel based on the first class classifying unit, and a class classified by the first class classifying unit, the target pixel and the target pixel being arranged spatially and / or temporally From multiple transmission pixelsLinear linear combination modelStandingBy least squaresConcernedLinear linear combination modelSolveAndParameter calculation means for temporarily storing the obtained coefficient as a parameter in the memory means, a parameter according to the class, and a plurality of transmission pixels arranged spatially and / or temporally around the target pixelLinear linear combination formula based onThe first estimated pixel value calculating means for calculating the first estimated pixel value of the pixel of interest by using and the difference value between the first estimated pixel value and the true pixel value included in the input image data is calculated. Difference value calculation means is provided, and the difference value is transmitted together with transmission pixel data and parameters.
[0022]
  In the present invention, the predetermined pixels of the input image data areOffsetTransmission pixel data in which the number of pixels is reduced by thinning out by sampling, and a plurality of transmissions arranged spatially and / or temporally with the pixel of interest as the pixel of interest, and the pixel of interest in the transmission pixel data as the center PixelGradation or patternEach pixel of interest is classified based on the classification, and for each classified class, the pixel of interest and a plurality of transmission pixels arranged spatially and / or temporally around the pixel of interestLinear linear combination modelStandingBy least squaresConcernedLinear linear combination modelSolveAndA parameter that is a coefficient obtained, a parameter according to a class, and a plurality of transmission pixels arranged spatially and / or temporally around the pixel of interestLinear linear combination formula based onIs an image signal transmission device that receives a difference value between a first estimated pixel value of a target pixel calculated by using and a true pixel value included in input image data, and is spatially centered on the target pixel. And / or multiple transmission pixels arranged in timeGradation or patternA second class classifying unit that classifies each target pixel based on the first class, a parameter according to the class classified by the second class classifying unit, and a plurality of spatially and / or temporally arranged with the target pixel as a center Transmission pixels andLinear linear combination formula based onThe second estimated pixel value calculating means for calculating the second estimated pixel value of the target pixel by using and the adding means for adding the difference value to the second estimated pixel value are provided.
[0023]
  Furthermore, in the present invention, predetermined pixels of the input image data areOffsetA first step of generating transmission pixel data with a reduced number of pixels by thinning out by sampling, and a pixel thinned out in the first step as a target pixel, and spatially and / or temporally with the target pixel as the center Multiple transmission pixels arrangedGradation or patternA second step for classifying each pixel of interest on the basis of the first step, and a plurality of transmission pixels arranged spatially and / or temporally with the pixel of interest as the center for each class classified in the second step And fromLinear linear combination modelStandingBy least squaresConcernedLinear linear combination modelSolveAndA third step of temporarily storing the obtained coefficient as a parameter in the memory means; a parameter corresponding to the class; and a plurality of transmission pixels arranged spatially and / or temporally with the pixel of interest at the centerLinear linear combination formula based onAnd a fifth step of calculating a difference value between the first estimated pixel value and the true pixel value included in the input image data. And a sixth step for transmitting the difference value together with the transmission pixel data and the parameter.
  Furthermore, in the present invention, predetermined pixels of the input image data areOffsetTransmission pixel data in which the number of pixels is reduced by thinning out by sampling, and a plurality of transmissions arranged spatially and / or temporally with the pixel of interest as the pixel of interest, and the pixel of interest in the transmission pixel data as the center PixelGradation or patternEach pixel of interest is classified based on the classification, and for each classified class, the pixel of interest and a plurality of transmission pixels arranged spatially and / or temporally around the pixel of interestLinear linear combination modelStandingBy least squaresConcernedLinear linear combination modelSolveAndA parameter that is a coefficient obtained, a parameter according to a class, and a plurality of transmission pixels arranged spatially and / or temporally around the pixel of interestLinear linear combination formula based onAn image signal transmission method for receiving a difference value between a first estimated pixel value of a target pixel calculated by using the true pixel value included in the input image data, and spatially centering on the target pixel And / or multiple transmission pixels arranged in timeGradation or patternA first step for classifying each pixel of interest on the basis of the parameters, a plurality of transmission pixels arranged spatially and / or temporally with the pixel of interest as a center and parameters according to the class classified in the first stepLinear linear combination formula based onIs used to provide a second step of calculating the second estimated pixel value of the target pixel and a third step of adding the difference value to the second estimated pixel value.
[0024]
[Action]
  A first estimated pixel value that will be obtained from the transmission pixel data and parameters on the receiving side is calculated in advance. Next, a difference value between the first estimated pixel value and the true pixel value included in the input image data is calculated, and this difference value is transmitted together with the transmission pixel data and parameters. As a result, the receiving side can obtain an interpolated pixel value substantially equal to the true value by canceling the error according to the difference value, thereby improving the resolution of the restored image.
[0025]
【Example】
Hereinafter, an embodiment of the present invention will be described in detail with reference to the drawings.
[0026]
(1) Overall configuration
In FIG. 1, reference numeral 40 denotes an image encoding apparatus to which the present invention is applied as a whole, and includes an encoded data forming unit 41 and an error data forming unit 42. The encoded data forming unit 41 generates compressed encoded data D2 from the input image data D1, and coefficient data D4 representing the correlation between the transmission pixels and the thinned pixels.
The error data forming unit 42 represents an error representing an error from the true value of the interpolated pixel value that will be generated on the receiving side based on the compressed encoded data D2 and the coefficient data D4 output from the encoded data forming unit 41. Data D20 is generated.
[0027]
In practice, the image encoding device 40 inputs the input image data D1 to the sub-sampling circuit 43 of the encoded data forming unit 41, where the information amount is reduced by thinning out predetermined pixels of the input image data D1, and then compression is performed. The data is sent to the encoder 44. The compression encoder 44 generates compression-encoded data D2 by compressing and encoding the remaining pixels after the thinning, and transmits this through the delay circuit 45.
[0028]
The output of the compression encoder 44 is decoded into data before compression coding via the local decoder 46, and the decoded data D 21 is given to the class classification circuit 47. The class classification circuit 47 uses the pixel to be interpolated (that is, the thinned pixel) as the target pixel, classifies each target pixel according to the state of the surrounding pixel in terms of space and / or time, and classifies the target pixel. The result is sent to the coefficient selection circuit 48 as class information (hereinafter referred to as index data) D22 of the target pixel. For example, the class classification circuit 47 classifies each target pixel according to the gradation and pattern of the transmission pixels around the target pixel.
[0029]
The coefficient selection circuit 48 is based on the input image data D1 and the decoded data D21 input through the delay circuit 49, and for each class, the pixel value of the pixel of interest included in the input image data D1 and the pixels of the surrounding decoded data D2 The correlation with the value is obtained by learning, and the learning result is output as coefficient data D4.
The delay circuit 49 delays the input image data D1 by the processing time required by the sub-sampling circuit 43, the compression encoder 44, and the local decoder 46.
[0030]
The error data forming unit 42 includes an interpolation data estimation circuit 50. The interpolation data estimation circuit 50 receives the decoded data D21 output from the local decoder 46 via the delay circuit 51 and is output from the coefficient selection circuit 48. Coefficient data D4 is received. The interpolation data estimation circuit 50 receives the index data D22 output from the class classification circuit 47 via the delay circuit 52. The delay circuit 51 delays the decoded data D21 by the processing of the class classification circuit 47, and the delay circuit 52 delays the index data D22 by the processing of the coefficient selection circuit 48.
[0031]
The interpolation data estimation circuit 50 generates in advance an interpolation pixel value that will be generated by using the compressed encoded data D2, coefficient data D4, and index data D22 on the receiving side, and estimated interpolation data obtained thereby. D23 is sent to the difference circuit 53.
The difference circuit 53 inputs the estimated interpolation data D23 and inputs the input image data D1 through the delay circuits 49 and 54, thereby calculating a difference between the estimated interpolation pixel value and the corresponding true interpolation pixel value. Do. As a result, the difference circuit 53 calculates an error value D24 representing an error of the estimated interpolation pixel value from the true interpolation pixel value. The error value D24 is quantized and encoded by the subsequent quantization circuit 55 and transmitted as error data D20.
[0032]
Incidentally, the delay time of the delay circuit 54 is selected so that the total delay time with the delay time of the delay circuit 49 is equal to the time required until the estimated interpolation pixel value is generated. The quantization circuit 55 is configured by, for example, an ADRC (Adaptive Dynamic Range Coding) circuit, and generates error data D20 by adaptively quantizing the error value D24.
In this way, the image encoding device 40 is configured to transmit the error data D20 from the error data forming circuit 42 together with the compression encoded data D2 and the coefficient data D4 from the encoded data forming circuit 41.
[0033]
The transmission data sent from the image encoding device 40 is received and decoded by an image decoding device 60 having a configuration as shown in FIG. The image decoding device 60 decodes the compressed encoded data D2 by the compression decoder 61, and sends the decoded data D30 obtained thereby to the class classification circuit 62.
The class classification circuit 62 has the same configuration as that of the class classification circuit 47 described above with reference to FIG. 1, and an interpolation target pixel that is not actually transmitted is set as a target pixel, and the spatially and / or temporally neighboring pixels of the target pixel. The interpolation target pixel is classified into classes according to the state. As a result, the class classification circuit 62 generates the index data D31 for each pixel of interest and sends it to the interpolation data estimation circuit 63.
[0034]
The interpolation data estimation circuit 63 also has the same configuration as the above-described interpolation data estimation circuit 50, and obtains estimated interpolation data D32 based on the index data D31, decoded data D30, and coefficient data D4 of each pixel to be interpolated (target pixel). . In practice, the interpolation data estimation circuit 63 selects coefficient data D4 corresponding to the index data D31, and uses the selected coefficient data D4 and the decoded data D30 around the pixel to be interpolated to establish a linear linear combination equation. Estimated interpolation data D32 is obtained.
[0035]
The estimated interpolation data D32 obtained in this way is sent to the adding circuit 64. The error data D20 is also input to the adder circuit 64 via the delay circuit 65 and the inverse quantization circuit 66. As a result, the addition circuit 64 cancels the error included in the estimated interpolation data D32, and generates interpolation data D33 equal to the true pixel value included in the input image data D1 (FIG. 1).
The time series conversion circuit 67 generates the restored image data D34 by arranging the decoded data D30 and the interpolation data D33 in the same manner as the original image (that is, the input image data D1).
Thus, the image decoding apparatus 60 can obtain an interpolated pixel substantially equal to the true value, thereby obtaining a restored image without image quality deterioration.
[0036]
(2) Detailed configuration
Next, detailed configurations of the sub-sampling circuit 43, the compression encoder 44, the compression decoder 61, the class classification circuits 47 and 62, the coefficient selection circuit 48, and the interpolation data estimation circuits 50 and 63 in the embodiment will be described.
[0037]
In this embodiment, as shown in FIG. 3, the sub-sampling circuit 43 performs a thinning process such that the number of transmission pixels is halved with respect to the frame image at each time point T0, T1, T2,. Apply. At this time, the sub-sampling circuit 43 selects a pixel position to be thinned out alternately so that pixels at the same position as the position sampled at the previous time point are not sampled at the next time point between successive time points. By performing spatial offset sub-sampling, the amount of transmitted information is reduced by half while leaving as much image features as possible.
3 represents a pixel remaining after thinning (that is, a transmission pixel), and a + mark represents a pixel to be interpolated (target pixel). The lower w₁~ W₃₈Represents a coefficient of a linear linear combination model obtained by a coefficient selection circuit 48 described later.
[0038]
The compression encoder 44 is composed of an ADRC (Adaptive Dynamic Range Coding) circuit. For example, pixel data having an information amount of 8 bits per pixel is obtained by calculating a difference between the maximum pixel value and the minimum pixel value in a predetermined block. The amount of transmission information is effectively reduced by expressing the dynamic range information to be expressed, the minimum pixel information, and the difference between each pixel value and the minimum pixel value, for example, by quantization information obtained by performing 1-bit quantization.
[0039]
  That is, as shown in FIG.Compression encoder 44) Divides the input data into minute blocks by the block forming circuit 70 and sends the pixel data in the block to the maximum / minimum value calculating circuit 71 for each minute block. The maximum value / minimum value calculation circuit 71 calculates the maximum value and the minimum value of the pixel values included in the minute block, and outputs the minimum pixel value MIN obtained thereby as transmission data. The maximum value and the minimum value are subtracted by the subtracting circuit 72, and the dynamic range information DR obtained thereby is output as transmission data.
[0040]
  Also, a difference value between each pixel value and the minimum pixel value in the minute block including the pixel value is calculated by the subtracting circuit 73, and the subtraction result is sent to the adaptive quantization circuit 74. The adaptive quantization circuit 74 sets the dynamic range DR to 2 when the number of quantization bits is n bits.ⁿQuantization step width is calculated by dividing by, and the subtraction result from the subtraction circuit 73 is divided by the quantization step width, and the division result obtained is rounded to an integer to generate a quantization code Q. In this way, the ADRC circuit (Compression encoder 44) Outputs dynamic range information DR, minimum value information MIN, and quantization code Q for each minute block as compressed encoded data D2.
[0041]
The compression decoder 61 (FIG. 2) corresponding to the compression encoder 44 is configured as shown in FIG. 4B, and inputs the dynamic range information DR and the quantization code Q to the adaptive inverse quantization circuit 75. Here, 2 for dynamic range DR for each blockⁿThe quantization step width is calculated by dividing by, and the quantization step width is multiplied by the quantization code Q. The compression decoder 61 adds the minimum value information MIN to the multiplication result by the addition circuit 76 and performs block decomposition by the subsequent block decomposition circuit 77 to obtain decoded data D30.
Incidentally, the local decoder 46 (FIG. 1) provided in the image encoding device 40 is also configured similarly to the compression decoder 61.
[0042]
The class classification circuits 47 and 62 are configured as shown in FIG. 5, and input the decoded data D21 (or D30) to the time series conversion circuit 80. The time-series conversion circuit 80 has, for example, a three-frame memory configuration, stores the decoded data D21 (or D30) for three frames that are temporally continuous, and the surrounding pixel values for each target pixel (that is, the interpolation target pixel). X1 to X14 are sent to subsequent subtraction circuits 81A to 81F.
[0043]
At this time, peripheral pixel values X1 to X14 are input to each of the subtraction circuits 81A to 81F so that the difference is obtained between the pixels arranged in the vertical direction, the horizontal direction, the diagonal direction, and the time direction with the target pixel as the center. Is done. Specifically, as shown in FIG. 6, the difference value between the pixels X1 and X2 arranged in the vertical direction is obtained by the subtraction circuit 81A, and the difference value between the pixels X3 and X4 in the horizontal direction is obtained by the subtraction circuit 81B. The difference value between the pixels X5 and X6 is subtracted by the subtracting circuit 81C, the difference value between the pixels X11 and X12 in the diagonal direction is subtracted by the subtracting circuit 81E, and the difference value between the pixels X13 and X14 in the time direction is subtracted by the subtracting circuit 81F. Is calculated by
6 represents the pixels (transmission pixels) left after thinning, among which ◎ represents the pixels used for class classification, and x represents the thinned pixels (non-transmission pixels). Represents.
[0044]
The difference value in each direction obtained as a result is given to the coding circuit 82, coded by the coding circuit 82 according to the difference value, and output as index data D22 (or D31).
In practice, the encoding circuit 82 is configured as shown in FIG. 7, and the absolute value calculation circuit 83 calculates the absolute value of each difference value and sends it to the threshold value determination circuit 84. The threshold value determination circuit 84 compares each difference absolute value with a predetermined threshold value, and when all the difference absolute values are less than the threshold value, sends a command signal S1 indicating this to the code allocation circuit 86. On the other hand, the threshold value determination circuit 84 sends all the difference absolute values to the minimum value detection circuit 85 when there are one or more difference absolute values equal to or greater than the threshold value.
[0045]
The minimum value detection circuit 85 detects the minimum value of the absolute difference values and sends the detection result to the code allocation circuit 86. The code allocation circuit 86 outputs a code corresponding to the direction in which the minimum value is detected. For example, as shown in the lower part of FIG. 6, code 0 is assigned when the difference absolute value between the pixels in the vertical direction is minimum, and code is assigned when the difference absolute value between the pixels in the horizontal direction is minimum. 1 is assigned, and when the difference absolute value between the pixels in the time direction is the smallest, code 6 is assigned. Seven types of codes are output according to the difference absolute values in the spatial direction and the time direction.
[0046]
Further, the code allocation circuit 86 outputs the code 7 when the command signal S1 is given from the threshold determination circuit 84. Here, the fact that the code allocation circuit 86 outputs the code 7 means that the periphery of the interpolation target pixel (target pixel) is a stationary flat portion.
As described above, in the class classification circuits 47 and 62 of the embodiment, the flatness of the peripheral pixels of the target pixel is detected, and the direction in which the flatness is the largest is output as the index data D22 (or D31). It is made to classify.
[0047]
The coefficient selection circuit 48 represents the interpolation target pixel by a linear linear combination model of the interpolation target pixel and a transmission pixel around the interpolation target pixel, and the coefficient used at this time is determined by the least square method for each class. Obtained by calculation.
The principle of coefficient selection will be described. As shown in FIG. 3, the interpolation target pixel y_mIf the frame in which T is present is T1, the interpolation target pixel y from this frame T1_mAnd a region around the same as the region cut out in the frame T1 from the frame T0 at the previous time point and the frame T2 at the next time point. Also, the interpolation target pixel y_mAre extracted from the input image data D1.
[0048]
Here, the coefficient w is first applied to each transmission pixel in the region cut out from the frames T0 to T2._iTo be interpolated pixel y_mIs represented by a linear linear combination of surrounding transmission pixels spatially and temporally. As a result, the interpolation target pixel y_mDeterminant Y and surrounding transmission pixels x_miThe determinant X is the coefficient w_iUsing the determinant W of
[Expression 1]

Can be expressed in the form of an observation equation. However, in the formula (1), n is one interpolation target pixel y_mRepresents the number of pixels to be transmitted spatially and temporally when represented by a linear linear combination formula (in the case of the embodiment, one interpolation target pixel y)._mIs represented by a 38-type linear linear combination model, so that n = 38), m represents the number of interpolation target pixels existing in one frame.
[0049]
Here, if one coefficient group is obtained for one frame based on the equation (1), simultaneous equations for the number of pixels to be interpolated in one frame are made from the equation (1). Basically, the coefficients can be obtained by solving the simultaneous equations. In this embodiment, the simultaneous equations are solved using a least square method.
That is, first, the equation (1) is expressed by the following equation using the residual matrix E:
[Expression 2]

Re-express it in the form of a residual equation like
[0050]
Here, each coefficient value w from equation (2)_iE to obtain the most probable value of₁ ²+ E₂ ²+ …… + e_m-1 ²+ E_m ²Which minimizes the following condition:
[Equation 3]

W satisfying this with n conditions₁, W₂, ..., w_nFind out.
Here, from the equation (2), the following equation:
[Expression 4]

If the conditions of equation (3) are set for i = 1, 2,..., N, respectively,
[Equation 5]

Is obtained. Here, the following normal equation is obtained from the equations (2) and (5).
[Formula 6]

[0051]
Here, since the normal equation represented by the equation (6) is a simultaneous equation with n unknowns, each coefficient w which is the most probable value is thereby obtained._iCan be requested. To be precise, w in equation (6)_i(Σx_jnx_jn) (Where j = 1... M) can be solved if the matrix is regular. Actually, simultaneous equations are solved using Gauss-Jordan elimination (sweeping method).
[0052]
In the case of the embodiment, for each class obtained by the class classification circuit 47, the coefficient value w is calculated using the least square method described above._iAsk for. As a result, it is only necessary to transmit one set of coefficients per frame for each class, and the coefficient values w for all interpolation target pixels._iThe amount of transmission information and the amount of computation can be significantly reduced as compared with the case where transmission is performed in response to the request. Actually, the information amount of the coefficient is on the order of negligible as compared with the pixel information amount per frame.
[0053]
Specifically, the coefficient selection circuit 48 is configured as shown in FIG. That is, the coefficient selection circuit 48 uses the locally decoded decoded data D21 (x_i) And interpolation target pixel data y included in the input image data D1_mIs input to the time series conversion memory 90. As described above with reference to FIG. 3, the time series conversion memory 90 has pixels (x) for forming a linear linear combination model.₁~ X_n, Y_m) Are output at the same time.
The data output from the time series conversion memory 90 is given to the normalization equation generation circuit 91, and the normalization equation generation circuit 91 generates a normalization equation as expressed by equation (6) for each class. Subsequently, the CPU arithmetic circuit 92 obtains a coefficient set for each class by the sweep-out method.
[0054]
The normalization equation generation circuit 91 first multiplies each pixel by the multiplier array 93. The multiplier array 93 is configured as shown in FIG. 9, performs multiplication of pixels for each cell represented by a square, and gives each multiplication result obtained thereby to the adder memory 94.
[0055]
As shown in FIG. 10, the adder memory 94 includes an adder array 95 composed of cells arranged similarly to the multiplier array 93 and memory (or register) arrays 96 </ b> A to 96 </ b> G. The memory arrays 96A to 96G are provided for the number of classes (in the example, seven classes excluding class 7 representing the stationary flat portion), and the output of the index decoder 97 for decoding the index data D22 ( In response to the class), one memory array 96A to 96F or 96G is selected, and the stored value of the selected memory array 96A to 96F or 96G is fed back to the adder array 95. At this time, the addition result obtained by the adder array 95 is stored again in the corresponding memory arrays 96A to 96F or 96G.
[0056]
In this way, the multiply-accumulate operation is performed by the multiplier array 93, the adder array 95, and the memory arrays 96A to 96G, and the memory arrays 96A to 96F or 96G are selected for each class determined by the index. The contents of the memory arrays 96A to 96G are updated according to the result of the product-sum operation.
[0057]
Note that the position of each array is represented by w in the normalization equation expressed by equation (6)._i(Σx_jnx_jn) (Where j = 1... M). As is clear from the normalization equation (6), if the upper right term is inverted, it becomes the same as the lower left, so each array has a triangular shape.
[0058]
In this way, a product-sum operation is performed for each class during one frame, and a normalization equation for each class is generated. The result of each term of the normalization equation for each class is stored in the memory arrays 96A to 96G corresponding to each class, and then each term of the normalization equation for each class realizes the sweep-out method operation. It is calculated by the CPU arithmetic circuit 92. As a result, the coefficient w for each class_iAre obtained and transmitted as coefficient data D4.
[0059]
As shown in FIG. 11, the interpolation data estimation circuits 50 and 63 have a coefficient memory 100 that stores a coefficient set of each class for each frame. The coefficient memory 100 is a coefficient set w for each class.₁~ W_nAnd a coefficient set w corresponding to the class in response to the output (class) of the index decoder 101.₁~ W_nIs output. This coefficient set w₁~ W_nAre respectively registers 102A.₁, 102A₂... 102A_nThrough the multiplier 103A₁, 103A₂... 103A_nGiven to. The multiplier 103A₁~ 103A_nIncludes the decoded data x selected by the time-series conversion circuit 104.₁~ X_nIs given. Therefore, the multiplier 103A₁~ 103A_nAre added by the adder circuit 105, so that the estimated value y (= x₁w₁+ X₂w₂+ …… + x_nw_n) Is obtained.
[0060]
(3) Operation of the embodiment
In the above configuration, the image encoding device 40 generates the compression encoded data D2 after reducing the data amount by performing the thinning process on the input image data D1. Further, the image encoding device 40 uses the thinned pixel as the target pixel, classifies each target pixel according to the state of the transmission pixel around the target pixel, and then obtains a coefficient set for each class, This is output as coefficient data D4.
[0061]
  Further, the image encoding device 40 calculates in advance the estimated interpolation data D23 that will be obtained on the image decoding device 60 side based on the compression encoded data D2 and the coefficient data D4 in the error data forming unit 42, and From the true value of the estimated interpolation data D23Error value D24CalculateError value D24The error data D20 based on is output.
[0062]
Here, since the coefficient selection circuit 48 of the image encoding device 40 generates coefficient data D4 by a kind of statistical processing, if an interpolation value is generated based on this coefficient data D4, the interpolation value and the true pixel value are inevitably generated. A certain amount of error will occur between the two. For this reason, if a restored image is generated using an interpolation value having an error, the image quality is degraded by this error.
In order to avoid this, the image encoding device 40 transmits error data D20 in addition to the compression encoded data D2 and the coefficient data D4.
[0063]
In the image decoding device 60, based on the decoded data D30 obtained by decoding the compression encoded data D2, the same class classification processing as that performed by the image encoding device 40 is performed, and the selection is made according to the index data D31 obtained thereby. The estimated interpolation data D32 is obtained by linearly combining the coefficient data D4 and the decoded data D30.
[0064]
Next, the image decoding apparatus 60 cancels the error included in the estimated interpolation data D32 by adding the inversely quantized data of the transmitted error code D20 to the estimated interpolation data D32.
Thus, since the interpolation data D33 having a value substantially equal to the actual value included in the input image data D1 can be obtained, a restored image with improved image quality can be obtained.
[0065]
(4) Effects of the embodiment
  According to the above configuration, the image encoding device 40 performs class classification processing, and image data(Compressed encoded data D2)Together with the coefficient data D4 for each class, and the image data transmitted by the image decoding device 60(Compressed encoded data D2)When the interpolation pixel is generated based on the coefficient data D4, the image encoding device 40 preliminarily estimates the interpolation pixel value(Estimated interpolation data D23)And true value(Input image data D1)Difference value from(Error data D20)The difference value(Error data D20)The image data(Compressed encoded data D2)And the coefficient data D4 are transmitted together, so that the image decoding apparatus 60 can restore a restored image with little image quality deterioration.(Restored image data D34)Can be obtained.
  This also increases the number of classes and restores images without increasing the amount of computation when selecting coefficients.(Restored image data D34)Image quality can be improved.
[0066]
(5) Other embodiments
In the above-described embodiment, the case where the ADRC circuit is used as the compression encoder 44 has been described. However, the present invention is not limited to this, and the compression encoder 44 may be, for example, DCT (Discrete Cosine Transform) transform coding, DPCM, vector quantum. The same effects as those of the above-described embodiment can be obtained even when a compression method such as conversion, subband coding, and wavelet transform is used.
[0067]
In the above-described embodiment, the case where the transmission pixel remaining after the thinning is compressed and transmitted by the compression encoder 44 has been described. However, the present invention is not limited to this, and the thinning remaining after the thinning without compression coding. The present invention can also be applied when data is transmitted as it is. In this case, the compression encoder 44 and the local decoder 46 may be omitted on the image encoding device 40 side, and the compression decoder 61 may be omitted on the image decoding device 60 side.
[0068]
In the above-described embodiment, the interpolation target pixel (target pixel) y_mIn the above description, the absolute value of the difference between the surrounding transmission pixels X1 to X14 is obtained, and the class classification is used in which the direction in which the difference absolute value is the smallest is the class of the pixel to be interpolated. Not limited to this, the point is that each pixel to be interpolated may be classified according to the state of the transmission pixel around the pixel to be interpolated, various class classification processes can be applied, and the number of classes can be varied. Can be used.
[0069]
Furthermore, if the transmission pixels used for class classification are once subjected to ADRC encoding and then class classification processing is performed, the number of class information can be reduced. In addition, if class classification is performed based on the output of the ADRC circuit in this way, accurate class classification based only on image characteristics can be performed even when the pixel level of the entire screen changes. .
[0070]
In the above-described embodiment, when the class classification circuit 47 is provided, the interpolation target pixel is classified according to the state of the surrounding transmission pixels, and the coefficient selection circuit 48 obtains a coefficient set for each class. However, the present invention is not limited to obtaining coefficient sets for each class. For example, an image is subdivided in space according to local characteristics of the image, and coefficient sets are obtained for each subdivided region. In this case, the class classification circuits 47 and 62 may be omitted. Thereby, for example, the present invention can also be applied to the image encoding device 1 and the image decoding device 10 configured as shown in FIG.
[0071]
In the above-described embodiment, coefficient data obtained by solving a linear linear combination model by the least square method as a parameter representing the correlation between the target pixel and the spatially and / or temporally surrounding transmission pixel of the target pixel. Although the case of transmitting D4 has been described, the present invention is not limited to this, and as a parameter, for example, an average value of transmission pixel values around the target pixel obtained for each class may be transmitted as a representative value. . In this case, if the centroid method is used for the average calculation, a representative value with a small error can be easily obtained for each class. In addition, the present invention is not limited to this, and various parameters that indicate the correlation between the target pixel that is not transmitted and the transmission pixel can be used.
[0072]
Further, in the above-described embodiment, the case where the sub-sampling circuit 43 performs the half-thinning process on the input image data D1 has been described. However, the present invention is not limited to this. Even when other thinning-out processing such as processing is performed, the same effect as in the above-described embodiment can be obtained.
[0073]
【The invention's effect】
  As described above, according to the present invention, the predetermined pixel of the input image data isOffsetSampling means for generating transmission pixel data in which the number of pixels is reduced by thinning out by sampling, and a pixel thinned out by the sampling means is a target pixel, and is arranged spatially and / or temporally around the target pixel. Multiple transmission pixelsGradation or patternA first class classifying unit that classifies each target pixel based on the first class classifying unit, and a class classified by the first class classifying unit, the target pixel and the target pixel being arranged spatially and / or temporally From multiple transmission pixelsLinear linear combination modelStandingBy least squaresConcernedLinear linear combination modelSolveAndParameter calculation means for temporarily storing the obtained coefficient as a parameter in the memory means, a parameter according to the class, and a plurality of transmission pixels arranged spatially and / or temporally around the target pixelLinear linear combination formula based onThe first estimated pixel value calculating means for calculating the first estimated pixel value of the pixel of interest by using and the difference value between the first estimated pixel value and the true pixel value included in the input image data is calculated. The difference value calculation means is provided, and the difference value is transmitted together with the transmission pixel data and the parameter, so that the pixels thinned out on the transmission side can be accurately interpolated on the reception side, and as a result, the resolution of the restored image is further improved. It can improve.
  As described above, according to the present invention, the predetermined pixels of the input image data areOffsetA first step of generating transmission pixel data with a reduced number of pixels by thinning out by sampling, and a pixel thinned out in the first step as a target pixel, and spatially and / or temporally with the target pixel as the center Multiple transmission pixels arrangedGradation or patternA second step for classifying each pixel of interest on the basis of the first step, and a plurality of transmission pixels arranged spatially and / or temporally with the pixel of interest as the center for each class classified in the second step And fromLinear linear combination modelStandingBy least squaresConcernedLinear linear combination modelSolveAndA third step of temporarily storing the obtained coefficient as a parameter in the memory means; a parameter corresponding to the class; and a plurality of transmission pixels arranged spatially and / or temporally with the pixel of interest at the centerLinear linear combination formula based onAnd a fifth step of calculating a difference value between the first estimated pixel value and the true pixel value included in the input image data. And a sixth step for transmitting the difference value together with the transmission pixel data and parameters.ByThe pixels thinned out on the transmission side can be accurately interpolated on the reception side, and as a result, the resolution of the restored image can be further improved.
[Brief description of the drawings]
FIG. 1 is a block diagram showing the overall configuration of an image encoding apparatus to which the present invention is applied.
FIG. 2 is a block diagram showing an overall configuration of an image decoding apparatus to which the present invention is applied.
FIG. 3 is a schematic diagram for explaining a thinning process by a sub-sampling circuit and a pixel and a coefficient used for a linear primary engagement model.
FIG. 4 is a block diagram for explaining an ADRC circuit used for a compression encoder and an ADRC decoding circuit used for a compression decoder and a local decoder.
FIG. 5 is a block diagram showing a configuration of a class classification circuit.
FIG. 6 is a schematic diagram for explanation of class classification.
FIG. 7 is a block diagram showing a configuration of a coding circuit.
FIG. 8 is a block diagram showing a configuration of a coefficient selection circuit.
FIG. 9 is a schematic diagram illustrating a configuration of a multiplier array.
FIG. 10 is a schematic diagram illustrating a configuration of an adder memory.
FIG. 11 is a block diagram showing a configuration of an interpolation data estimation circuit.
FIG. 12 is a block diagram showing a configuration of a conventional digital data converter.
FIG. 13 is a block diagram showing a configuration of a conventional digital data conversion apparatus.
[Explanation of symbols]
40... Image encoding device, 41... Encoded data forming unit, 42... Error data forming unit, 43... Subsampling circuit, 44. Classification circuit 48... Coefficient selection circuit 50, 63... Interpolation data estimation circuit 53. Difference circuit 60 60 Image decoding device 64 Adder circuit D 1 Input image data D 2. Compression coded data, D4... Coefficient data, D20... Error data, D22, D31... Index data, D23, D32. Restored image data, W₁~ W₃₈……coefficient.

Claims (10)

Sampling means for generating transmission pixel data with a reduced number of pixels by thinning out predetermined pixels of the input image data by offset sampling;
A pixel thinned out by the sampling means is set as a pixel of interest, and each pixel of interest is classified based on gradations or patterns of a plurality of the transmission pixels arranged spatially and / or temporally around the pixel of interest. A first class classification means;
For each classified class by the first classification means, making a linear combination model from the plurality of the transmission pixel spatially and or temporally arranged about said target pixel and the target pixel, the minimum the coefficients obtained had a solution the linear combination model by calculating the square method as a parameter, a parameter calculating means for temporarily stored in the memory means,
A first estimated pixel value of the target pixel by using a linear linear combination formula based on the parameter according to the class and a plurality of the transmission pixels arranged spatially and / or temporally around the target pixel First estimated pixel value calculating means for calculating
Difference value calculation means for calculating a difference value between the first estimated pixel value and a true pixel value included in the input image data,
The image signal transmission device, wherein the difference value is transmitted together with the transmission pixel data and the parameter. The first class classification means includes:
By calculating a difference between the transmission pixel in the vertical direction, the transmission pixel in the horizontal direction, the transmission pixel in the diagonal direction, and the transmission pixel in the time direction at the same pixel position, which are arranged around the pixel of interest. The flatness of the transmission pixel in the vertical direction, the horizontal direction, the diagonal direction, and the time direction at the same pixel position is detected, and the pixel of interest is classified based on the flatness. 2. The image signal transmission device according to 1. The image signal transmission apparatus according to claim 1, further comprising an encoding unit that encodes the difference value obtained by the difference value calculation unit, and encoding and transmitting the difference value. Transmission pixel data in which the number of pixels is reduced by thinning out predetermined pixels of the input image data by offset sampling,
The thinned pixel in the transmission pixel data is a pixel of interest, and each pixel of interest is classified based on the gradations or patterns of the plurality of transmission pixels arranged spatially and / or temporally around the pixel of interest. It is, making a linear combination model from the plurality of the transmission pixel spatially and or temporally arranged in each classified class about said target pixel and the target pixel, the linear by calculation of the minimum square method and the parameter is a coefficient obtained by have solutions linear combination model,
The first pixel of the target pixel calculated by using a linear linear combination formula based on the parameter according to the class and a plurality of the transmission pixels arranged spatially and / or temporally around the target pixel. An image signal transmission device that receives a difference value between an estimated pixel value and a true pixel value included in the input image data,
Second class classification means for classifying each pixel of interest based on gradations or patterns of the plurality of transmission pixels arranged spatially and / or temporally around the pixel of interest;
By using a linear linear combination formula based on the parameters according to the class classified by the second class classification means and the plurality of transmission pixels arranged spatially and / or temporally around the pixel of interest. Second estimated pixel value calculating means for calculating a second estimated pixel value of the target pixel;
An image signal transmission apparatus comprising: addition means for adding the difference value to the second estimated pixel value. The difference value is compression encoded data,
The image signal transmission apparatus according to claim 4, further comprising decoding means for decoding the compression-encoded difference value. A first step of generating transmission pixel data with a reduced number of pixels by thinning out predetermined pixels of the input image data by offset sampling;
The pixel thinned out in the first step is set as the target pixel, and each target pixel is classified based on the gradation or pattern of the plurality of transmission pixels arranged spatially and / or temporally around the target pixel. A second step;
For each class classified in the second step, a linear linear combination model is established from the pixel of interest and a plurality of the transmission pixels arranged spatially and / or temporally around the pixel of interest . the coefficients obtained had a solution the linear combination model by calculation as a parameter, and the third step to be temporarily stored in the memory means,
A first estimated pixel value of the target pixel by using a linear linear combination formula based on the parameter according to the class and a plurality of the transmission pixels arranged spatially and / or temporally around the target pixel A fourth step of calculating
A fifth step of calculating a difference value between the first estimated pixel value and a true pixel value included in the input image data;
An image signal transmission method comprising: a sixth step of transmitting the difference value together with the transmission pixel data and the parameter. In the second step,
To calculate the difference between the transmission pixel in the vertical direction, the transmission pixel in the horizontal direction, the transmission pixel in the diagonal direction, and the transmission pixel data in the time direction at the same pixel position, each arranged around the pixel of interest. Therefore, the flatness of the transmission pixel in the vertical direction, the horizontal direction, the diagonal direction, and the time direction at the same pixel position is detected, and the pixel of interest is classified based on the flatness. Item 7. The image signal transmission method according to Item 6. In the sixth step,
The image signal transmission method according to claim 6, wherein the difference value obtained in the fifth step is encoded and transmitted. Transmission pixel data in which the number of pixels is reduced by thinning out predetermined pixels of the input image data by offset sampling,
The thinned pixel in the transmission pixel data is a pixel of interest, and each pixel of interest is classified based on the gradations or patterns of the plurality of transmission pixels arranged spatially and / or temporally around the pixel of interest. It is, making a linear combination model from the plurality of the transmission pixel spatially and or temporally arranged in each classified class about said target pixel and the target pixel, the linear by calculation of the minimum square method and the parameter is a coefficient obtained by have solutions linear combination model,
The first pixel of the target pixel calculated by using a linear linear combination formula based on the parameter according to the class and a plurality of the transmission pixels arranged spatially and / or temporally around the target pixel. An image signal transmission method for receiving a difference value between an estimated pixel value and a true pixel value included in the input image data,
A first step of classifying each pixel of interest based on gradations or patterns of the plurality of transmission pixels arranged spatially and / or temporally around the pixel of interest;
The pixel of interest by using a linear linear combination formula based on the parameter according to the class classified in the first step and a plurality of the transmission pixels arranged spatially and / or temporally around the pixel of interest. A second step of calculating a second estimated pixel value of
And a third step of adding the difference value to the second estimated pixel value. The difference value is compression encoded data,
14. The image signal transmission method according to claim 13, wherein in the third step, the compression-coded difference value is decoded.

Images