JP4415468B2 - Interpolation processing apparatus and recording medium recording interpolation processing program - Google Patents

Description

ただし、GvL,GvR,GhL,GhRは、以下の《平均補間》または《色差補間》のようにして算出される値である。
【０１２７】
《平均補間》
GvL=(G[i,j-1]+G[i,j+1])/2 ・・・式41
GvR=(G[i+1,j-1]+G[i+1,j+1])/2 ・・・式42
GhL=(2・G[i-1,j]+G[i+2,j])/3 ・・・式43
GhR=(G[i-1,j]+2・G[i+2,j])/3 ・・・式44
すなわち、《平均補間》において、GvL,GvRは、各々の補間対象サブピクセルの縦方向に隣接して緑色成分が存在するサブピクセルの色情報の平均値である。一方、GhL,GhRは、各々の補間対象サブピクセルの横方向に近接して緑色成分が存在するサブピクセルの色情報を、各々の補間対象サブピクセルとの距離に応じて平均化した値である。
【０１２８】
《色差補間》
GvL=(G[i,j-1]+G[i,j+1])/2+(2・Z[i,j]-Z[i,j-2]-Z[i,j+2])/4 ・・・式45
GvR=(G[i+1,j-1]+G[i+1,j+1])/2
+(2・Z[i+1,j]-Z[i+1,j-2]-Z[i+1,j+2])/4 ・・・式46
GhL=(2・G[i-1,j]+G[i+2,j])/3
+(5・Z[i,j]-2・Z[i-3,j]-2・Z[i+1,j]-Z[i+4,j])/9 ・・・式47
GhR=(G[i-1,j]+2・G[i+2,j])/3
+(5・Z[i+1,j]-2・Z[i+4,j]-2・Z[i,j]-Z[i-3,j])/9 ・・・式48
すなわち、《色差補間》では、緑色成分と補間対象サブピクセルにおける色成分（赤色成分または青色成分）との色差（G-RやG-Bに相当する）が一定であると仮定し、補間対象サブピクセルの縦方向（または、横方向）に近接する緑色成分が存在するサブピクセルの色成分（赤色成分または青色成分）の補間とその位置における色差の成分の算出、および、このような色差を用いた補間対象サブピクセルの色差の補間、補間対象サブピクセルの色差を緑の補間値に変換する演算を一括して行うことに相当する。
【０１２９】
また、式45〜式48は、《平均補間》の式41〜式44と同様にして算出される値を、補間対象サブピクセルと同色の色情報を用いて補正することによって、緑の補間値を算出する演算式であると言える。すなわち、式45〜式48において、第１項は、「緑色成分の局所的な平均情報」であり、第２項は、補間対象サブピクセルにおける凹凸の情報を反映した「補間対象サブピクセルと同一の色成分による局所的な変曲情報」である。
【０１３０】
ただし、補間処理が行われる前の画像データに対してγ階調変換を施す場合、γ変換が対数特性に近いので、《色差補間》では、色成分の比（G/RやG/Bに相当する）が一定であると仮定することになる。
なお、図６は、《色差補間》で用いる色情報の位置を示す図である。
以上説明したように、補間処理部１７は、主要類似度によって画素ピッチでの類似性の強弱を判定し、このような判定によって類似性が強い方向が不明であった場合に、微細部類似度によってサブピクセルピッチでの類似性の強弱を判定するが、主要類似度および微細部類似度は、補間対象画素の類似度成分と周辺画素の類似度成分とを加重加算することによって算出されるため、補間対象画素と周辺画素とにおける色情報の連続性が反映され易く、広域的な判定をすることができる。また、微細部類似度の算出時には、複数種類の微細部類似度成分が加重加算されるため、多くの色情報が用いられることになり、このような加重加算時には、複数種類の微細部類似度成分がサブピクセル間の距離によって規格化される。
【０１３１】
したがって、第１の実施形態のＧ補間処理では、補間対象画素との類似性が強い方向を精度良く判定することができるため、画像全体の調和を崩すことなく補間値を算出することが可能である。
次に、補間処理部１７は、ＲＢ補間処理を行う。
以下、ＲＢ補間処理の動作として、図３に示した《ＲＢ補間処理１》と《ＲＢ補間処理２》との２つのＲＢ補間処理の動作を説明する。なお、以下では、赤の補間値（または、青の補間値）を補うべき２つのサブピクセルを、補間対象サブピクセルと称して説明を行う。また、青の補間値は、赤の補間値と同様に算出できるため、以下では、説明を省略する。
【０１３２】
《ＲＢ補間処理１》
まず、《ＲＢ補間処理１》において、補間処理部１７は、補間対象サブピクセルに欠落する色成分が存在するサブピクセル毎に、その色成分の色情報から緑の補間値（上述したＧ補間処理によって得られた値）を減算し、色差を算出する（図３Ｓ１０）。
【０１３３】
例えば、補間処理部１７は、赤色成分の色情報が存在して互いに隣接する２つのサブピクセルの座標を[i,j],[i+1,j]とすると、これらのサブピクセルにおける赤色成分を含む色差Cr[i,j],Cr[i,j]を、
Cr[i,j]=R[i,j]-G[i,j] ・・・式49
Cr[i+1,j]=R[i+1,j]-G[i+1,j] ・・・式50
によって算出する。
【０１３４】
次に、補間処理部１７は、補間対象サブピクセルに対し最も近接するサブピクセルで算出された色差を用いて、補間対象サブピクセルの色差を線形補間する（図３Ｓ１１）。
例えば、２つの補間対象サブピクセルの座標を[m,n],[m+1,n]とし、これらの２つの補間対象サブピクセルに赤色成分の色情報が欠落していると仮定すると、補間処理部１７は、座標[m,n],[m+1,n]の垂直方向に位置するサブピクセルで赤色成分を含む色差が算出されている場合（図７（１）に対応する）には、赤色成分を含む色差Cr[m,n],Cr[m+1,n]を、
Cr[m,n]=(Cr[m,n-1]+Cr[m,n+1])/2 ・・・式51
Cr[m+1,n]=(Cr[m+1,n-1]+Cr[m+1,n+1])/2 ・・・式52
によって線形補間する。
【０１３５】
また、補間処理部１７は、座標[m,n],[m+1,n]の水平方向に位置するサブピクセルで赤色成分を含む色差が算出されている場合（図７（２）に対応する）には、赤色成分を含む色差Cr[m,n],Cr[m+1,n]を、
Cr[m,n]=(2・Cr[m-1,n]+Cr[m+2,n])/3 ・・・式53
Cr[m+1,n]=(Cr[m-1,n]+2・Cr[m+2,n])/3 ・・・式54
によって線形補間する。
【０１３６】
さらに、補間処理部１７は、座標[m,n],[m+1,n]の斜め方向に位置するサブピクセルで赤色成分を含む色差が算出されている場合（図７（３）に対応する）には、赤色成分を含む色差Cr[m,n],Cr[m+1,n]を、
Cr[m,n]=(2・(Cr[m-1,n-1]+Cr[m-1,n+1])
+Cr[m+2,n-1]+Cr[m+2,n+1])/6 ・・・式55
Cr[m+1,n]=(Cr[m-1,n-1]+Cr[m-1,n+1]
+2・(Cr[m+2,n-1]+Cr[m+2,n+1]))/6 ・・・式56
によって線形補間する。
【０１３７】
次に、補間処理部１７は、補間対象サブピクセル毎に、緑色成分の色情報（緑色の補間値であっても良い）によって色差を補間値に変換する（図３Ｓ１２）。すなわち、補間処理部１７は、上述した任意の座標[m,n],[m+1,n]に位置するサブピクセルの赤の補間値R[m,n],R[m+1,n]については、
R[m,n]=Cr[m,n]+G[m,n] ・・・式57
R[m+1,n]=Cr[m+1,n]+G[m+1,n] ・・・式58
によって算出する。
【０１３８】
ところで、《ＲＢ補間処理１》では、孤立点（周辺画素との類似性の弱い空間周波数の高い画像部分）の周辺に偽色が残るという問題が生じるため、《ＲＢ補間処理１》を行った後には、画像データをＬａｂ表色系に変換して得られる色相ａ，ｂ面に対して個別にmedianフィルタを掛けるという事後処理を行うことがある。
【０１３９】
また、このようにmedianフィルタを掛ける場合、ベイア配列上の３×３（＝９点）のフィルタサイズでは、ほとんど有効に作用しないため、フィルタサイズをベイア配列上の５×５（＝２５点）の広域に設定する必要がある。
すなわち、《ＲＢ補間処理１》によって孤立点の周辺の偽色を取り除いた赤や青の補間値を得るには、フィルタサイズが広域に設定されたmedian処理を別途行う必要があるため、非常に重い処理が課せられることになる。また、median処理におけるフィルタサイズを大きくすることは、彩色部分の細かい構造（以下、「色構造」と称する）を破壊する危険性を高めることに繋がる。
【０１４０】
そこで、以下では、《ＲＢ補間処理２》として、孤立点の周辺に偽色を残すことなく、赤や青の補間値を高速かつ高精度で算出できるＲＢ補間処理を提示する。なお、《ＲＢ補間処理２》は、請求項５３ないし請求項５８に対応する。
《ＲＢ補間処理２》
まず、《ＲＢ補間処理２》において、補間処理部１７は、《ＲＢ補間処理１》と同様に、補間対象サブピクセルに欠落する色成分が存在するサブピクセル毎に色差を算出する（図３Ｓ２０）。
【０１４１】
なお、このような処理によって、赤色成分を含む色差が算出された状態では、赤色成分を含む色差は、赤色成分の色情報が欠落して青色成分の色情報が存在する２つのサブピクセルを斜め４方向から囲むように配置されることになる。
補間処理部１７は、色差が算出されたサブピクセルによって斜め４方向が囲まれる補間対象サブピクセル毎に、斜め方向に位置するサブピクセルにおける色差の中央値で補間対象サブピクセルの色差を補間する（図３Ｓ２１）。
【０１４２】
例えば、図８（１）に示すように、赤色成分を含む色差Crが算出されたサブピクセルによって、斜め４方向が囲まれる任意の２つのサブピクセルの座標を[m,n],[m+1,n]とすると、補間処理部１７は、これら２つサブピクセルにおける赤色を含む色差Cr[m,n],Cr[m+1,n]を、
Cr[m,n]=median｛Cr[m-1,n-1],Cr[m-1,n+1],
Cr[m+2,n-1],Cr[m+2,n+1]｝ ・・・式59
Cr[m+1,n]=median｛Cr[m-1,n-1],Cr[m-1,n+1],
Cr[m+2,n-1],Cr[m+2,n+1]｝ ・・・式60
によって算出する。ただし、median｛ ｝は、複数の要素のmedian値を算出する関数を示し、要素の数が偶数である場合、真ん中２つの要素の平均値をとるものとする。
【０１４３】
なお、以上説明したようにして赤色成分を含む色差が算出された状態では、赤色成分を含む色差は、赤色成分および青色成分の色情報が欠落する２つのサブピクセルを上下左右４方向から囲むように配置されることになる。
補間処理部１７は、色差が算出されたサブピクセルによって上下左右４方向が囲まれる補間対象サブピクセル毎に、上下左右方向に位置するサブピクセルにおける色差の中央値で補間対象サブピクセルの色差を補間する（図３Ｓ２２）。
【０１４４】
例えば、図８（２）に示すように、赤色成分を含む色差Crが算出されたサブピクセルによって、上下左右４方向が囲まれる任意の２つのサブピクセルの座標を[m,n],[m+1,n]とすると、補間処理部１７は、これら２つサブピクセルにおける赤色を含む色差Cr[m,n],Cr[m+1,n]を、
Cr[m,n]=median｛Cr[m-1,n],Cr[m+2,n],
Cr[m,n-1],Cr[m,n+1]｝ ・・・式61
Cr[m+1,n]=median｛Cr[m-1,n],Cr[m+2,n],
Cr[m+1,n-1],Cr[m+1,n+1]｝ ・・・式62
によって算出する。
【０１４５】
次に、補間処理部１７は、《ＲＢ補間処理１》と同様に、補間対象サブピクセル毎に、緑色成分の色情報によって色差を補間値に変換する（図３Ｓ２３）。
以上説明したように、《ＲＢ補間処理２》では、median処理は、色相を表す色差に対してのみ行われ、輝度成分に対しては行われない。また、ＲＢ補間処理において、図９（１）の○印が付与された２つのサブピクセルに赤の補間値を補う場合、ベイア配列上の３×５の範囲に位置する色差Crを用いて×印が付与されたサブピクセルにおける赤色成分を含む色差が算出されるので、○印が付与された２つのサブピクセルにおける赤色成分を含む色差は、ベイア配列上の３×５の範囲に位置する色差Crに重み付けをしてmedian処理を行った結果に近い値を示す。また、図９（２）の△印が付与された２つのサブピクセルに赤の補間値を補う場合、ベイア配列上の５×３の範囲に位置する色差Crを用いて×印が付与されたサブピクセルにおける赤色成分を含む色差が算出されるので、△印が付与された２つのサブピクセルにおける赤色成分を含む色差は、ベイア配列上の５×３の範囲に位置する色差Crに重み付けをしてmedian処理を行った結果に近い値を示す。
【０１４６】
すなわち、《ＲＢ補間処理２》では、フィルタサイズを小さく抑えつつ、実質的には広域なmedian処理と同等の効果が得られるため、色構造を破壊することなく、孤立点の周辺における偽色の発生を低減することができる。
また、《ＲＢ補間処理２》では、補間処理を行いながらmedian処理を行うため、《ＲＢ補間処理１》を行った後にmedian処理を行う場合と比べて処理効率が良く、かつ、図３Ｓ２１と図３Ｓ２２との各々のmedian処理において、４点の色差のみを用いるため、非常に高速なmedian処理が可能である。
【０１４７】
したがって、《ＲＢ補間処理２》では、色構造を破壊することなく、孤立点の周辺における偽色の発生を低減しつつ、補間値を高速に算出することができる。
ところで、上述した第１の実施形態におけるＧ補間処理では、色差が一定でない色境界部分において、偽色が発生し易い。
また、撮影光学系１２に倍率色収差が存在する場合、赤色成分および青色成分は、図１０に示すように、緑色成分の結像位置から、それぞれ少しずつずれた位置に結像する（以下、このような現象を「色ズレ」と称する。）。そのため、第１の実施形態では、このような色ズレ等によって色差が適切に算出できず偽色が発生する可能性がある。特に、《色差補間》を適用したＧ補間処理では、後述する倍率色収差による過補正が原因で偽色が発生するおそれがある。
【０１４８】
図１１は、倍率収差による過補正を説明する図である。
図１１では、撮影光学系１２に倍率色収差が存在しない状態で、各サブピクセルの色情報が図１１（１）の●印のように与えられている場合（補間処理の対象となる画像データの類似性が水平方向に強く、緑色成分の色情報が一定で、赤色成分や青色成分の色情報が補間対象サブピクセル（Ｚ５、Ｚ６が存在するサブピクセル）の近傍でなだらかに変化する場合）、式47や式48の第２項が０となるため、Ｇ４とＧ７とを平均化した値（式47や式48の第１項に相当する）は補正されることなくＧ５やＧ６（緑の補間値）となる。
【０１４９】
しかし、Ｚ１、Ｚ２、Ｚ５、Ｚ６、Ｚ９、Ｚ１０が赤色成分の色情報であり、撮影光学系１２の倍率色収差によって、これらの赤色成分の色情報が右方向に２サブピクセル分ずつずれた場合、各サブピクセルの色情報は、図１１（２）のように変化することになる。そのため、このような場合、式47や式48の第２項が０とならず、Ｇ４とＧ７とを平均化した値が過剰に補正（以下、「過補正」と称する）され、本来、Ｇ４やＧ７と同程度の値となるべきＧ５やＧ６は、Ｇ４やＧ７よりも大きな値となる。
【０１５０】
なお、Ｚ１、Ｚ２、Ｚ５、Ｚ６、Ｚ９、Ｚ１０が青色成分の色情報であり、撮影光学系１２の倍率色収差によって、これらの青色成分の色情報が左方向に２サブピクセル分ずつずれた場合、本来、Ｇ４やＧ７と同程度の値となるべきＧ５やＧ６は、Ｇ４やＧ７よりも小さい値となる。
そこで、第２の実施形態では、このような過補正等による偽色の発生を抑制できる補間処理を提示する。
【０１５１】
《第２の実施形態》
以下、第２の実施形態の動作を説明する。なお、第２の実施形態におけるＲＢ補間処理は、第１の実施形態と同様にして行えるため、ここでは、説明を省略する。
図１２および図１３は、第２の実施形態における補間処理部１７の動作フローチャートであり、Ｇ補間処理における補間処理部１７の動作を示す。
【０１５２】
以下、第２の実施形態におけるＧ補間処理について説明するが、第１の実施形態と同じ動作については、詳細な説明を省略する。
まず、補間処理部１７は、第１の実施形態と同様に、縦方向の主要類似度Cv[i_p,j]と、横方向の主要類似度Ch[i_p,j]とを算出して（図１２Ｓ１）、縦横主要類似性を比較し（図１２Ｓ２）、指標HV[i_p,j]に値を設定する（図１２Ｓ３〜Ｓ５）。
【０１５３】
次に、補間処理部１７は、ベイア配列上の補間対象画素における斜め45度方向の主要類似度C45[i_p,j]および斜め135度方向の主要類似度C135[i_p,j]を算出する（図１２Ｓ６）。
ここで、斜め45度方向の類似度C45[i_p,j]および斜め135度方向の類似度C135[i_p,j]を算出する処理の詳細を説明する。
【０１５４】
まず、補間処理部１７は、以下の式63〜式70によって定義される斜め45度方向および斜め135度方向に対する複数種類の主要類似度成分を算出する。
斜め45度方向のGG間主要類似度成分：
C45_1[i_p,j]=(|G[i_p,j-1]-G[i_p-1,j]|+|G[i_p+1,j]-G[i_p,j+1]|)/2 ・・・式63
斜め135度方向のGG間主要類似度成分：
C135_1[i_p,j]=(|G[i_p,j-1]-G[i_p+1,j]|+|G[i_p-1,j]-G[i_p,j+1]|)/2 ・・・式64
斜め45度方向のBB(RR)間主要類似度成分：
C45_2[i_p,j]=|Z[i_p+1,j-1]-Z[i_p-1,j+1]| ・・・式65
斜め135度方向のBB(RR)間主要類似度成分：
C135_2[i_p,j]=|Z[i_p-1,j-1]-Z[i_p+1,j+1]| ・・・式66
斜め45度方向のRR(BB)間主要類似度成分：
C45_3[i_p,j]=(|Z[i_p+2,j-2]-Z[i_p,j]|+|Z[i_p-2,j+2]-Z[i_p,j]|)/2 ・・・式67
斜め135度方向のRR(BB)間主要類似度成分：
C135_3[i_p,j]=(|Z[i_p-2,j-2]-Z[i_p,j]|+|Z[i_p+2,j+2]-Z[i_p,j]|)/2 ・・・式68
斜め45度方向のBR(RB)間主要類似度成分：
C45_4[i_p,j]=(|Z[i_p+1,j-1]-Z[i_p,j]|+|Z[i_p-1,j+1]-Z[i_p,j]|)/2 ・・・式69
斜め135度方向のBR(RB)間主要類似度成分：
C135_4[i_p,j]=(|Z[i_p-1,j-1]-Z[i_p,j]|+|Z[i_p+1,j+1]-Z[i_p,j]|)/2 ・・・式70
なお、上述した各々の主要類似度成分を構成する類似度の要素は、差分の絶対値を用いて算出しているが、絶対値の２乗やべき乗等によって算出しても良い。
【０１５５】
次に、補間処理部１７は、以下の式71および式72に示すようにして、加重係数b1,b2,b3,b4によって、複数種類の主要類似度成分を各々の方向毎に加重加算する。
C45_0[i_p,j]=(b1・C45_1[i_p,j]+b2・C45_2[i_p,j]+b3・C45_3[i_p,j]
+b4・C45_4[i_p,j])/(b1+b2+b3+b4) ・・・式71
C135_0[i_p,j]=(b1・C135_1[i_p,j]+b2・C135_2[i_p,j]+b3・C135_3[i_p,j]
+b4・C135_4[i_p,j])/(b1+b2+b3+b4) ・・・式72
なお、式71および式72において、加重係数b1,b2,b3,b4の比率としては、例えば、「b1:b2:b3:b4=2:1:1:2」などが考えられる。
【０１５６】
このようにして算出されるC45_0[i_p,j]およびC135_0[i_p,j]は、そのまま、補間対象画素の斜め45度方向の主要類似度C45[i_p,j]および斜め135度方向の主要類似度C135[i_p,j]とすることができるが、ここでは、周辺加算によって、これらの類似度を算出する例を示す。
すなわち、補間処理部１７は、補間対象画素と周辺画素とにおける主要類似度成分の方向別の加重加算の結果（C45_0[i_p,j]、C45_0[i_p-1,j-1]、C45_0[i_p-1,j+1]、C45_0[i_p+1,j-1]、C45_0[i_p+1,j+1]など）を、以下の《周辺加算方法１》または《周辺加算方法２》のように加重加算して、補間対象画素の斜め45度方向の主要類似度C45[i_p,j]および斜め135度方向の主要類似度C135[i_p,j]を算出する（図４に示すように補間対象画素と周辺画素とにおける類似度成分を加重加算することに相当する）。
【０１５７】
《周辺加算方法１》
C45[i_p,j]=(4・C45_0[i_p,j]+C45_0[i_p-1,j-1]+C45_0[i_p+1,j-1]
+C45_0[i_p-1,j+1]+C45_0[i_p+1,j+1])/8 ・・・式73
C135[i_p,j]=(4・C135_0[i_p,j]+C135_0[i_p-1,j-1]+C135_0[i_p+1,j-1]
+C135_0[i_p-1,j+1]+C135_0[i_p+1,j+1])/8 ・・・式74
《周辺加算方法２》
C45[i_p,j]=(4・C45_0[i_p,j]+2・(C45_0[i_p-1,j-1]+C45_0[i_p+1,j-1]
+C45_0[i_p-1,j+1]+C45_0[i_p+1,j+1])+C45_0[i_p,j-2]
+C45_0[i_p,j+2]+C45_0[i_p-2,j]+C45_0[i_p+2,j])/16 ・・・式75
C135[i_p,j]=(4・C135_0[i_p,j]+2・(C135_0[i_p-1,j-1]+C135_0[i_p+1,j-1]
+C135_0[i_p-1,j+1]+C135_0[i_p+1,j+1])+C135_0[i_p,j-2]
+C135_0[i_p,j+2]+C135_0[i_p-2,j]+C135_0[i_p+2,j])/16 ・・・式76
なお、このようにして算出される斜め45度方向の主要類似度C45[i_p,j]および斜め135度方向の主要類似度C135[i_p,j]において、複数種類の主要類似度成分や周辺画素との加重加算は、上述した縦方向の主要類似度Cv[i_p,j]および横方向の主要類似度Ch[i_p,j]と同様の役割を果たす。また、第２の実施形態において、斜め45度方向の主要類似度C45[i_p,j]および斜め135度方向の主要類似度C135[i_p,j]は、値が小さい程、類似性が強いことを示す。
【０１５８】
補間処理部１７は、斜め45度方向の主要類似度C45[i_p,j]および斜め135度方向の主要類似度C135[i_p,j]を算出すると、斜め45度方向の主要類似度C45[i_p,j]および斜め135度方向の主要類似度C135[i_p,j]に基づき、補間対象画素の斜め45度方向および斜め135度方向に対する類似性（以下、「斜め主要類似性」と称する。）を判定する（図１２Ｓ７）。そして、このような判定結果として、斜め方向の類似性を示す指標DN[i_p,j]に以下のような値を設定する。
【０１５９】
例えば、補間処理部１７は、任意の閾値Th2について、
|C45[i_p,j]-C135[i_p,j]|＞Th2 かつ C45[i_p,j]＜C135[i_p,j]
が成り立つ場合、斜め135度方向よりも斜め45度方向の主要類似性が強いと判断して指標DN[i_p,j]に１を設定し（図１２Ｓ８）、
|C45[i_p,j]-C135[i_p,j]|＞Th2 かつ C45[i_p,j]＞C135[i_p,j]
が成り立つ場合、斜め45度方向よりも斜め135度方向の主要類似性が強いと判断して指標DN[i_p,j]に−１を設定し（図１２Ｓ９）、
|C45[i_p,j]-C135[i_p,j]|≦Th2
が成り立つ場合、斜め方向間で主要類似性に区別がつかないと判断して指標DN[i_p,j]に０を設定する（図１２Ｓ１０）。
【０１６０】
なお、閾値Th2は、第１の実施形態における縦横主要類似性の比較に用いた閾値Th1と同様に、ノイズの影響によって一方の類似性が強いと誤判定されることを避ける役割を果たす。
次に、補間処理部１７は、以上説明したような縦横主要類似性および斜め主要類似性の判定において、主要類似性が強い方向が不明であったか否かを判定し（図１２Ｓ１１）、主要類似性が強い方向が不明であった場合には、第１の実施形態と同様に、縦方向の微細部類似度Cvs[i_p,j]と横方向の微細部類似度Chs[i_p,j]とを算出する（図１２Ｓ１２）。
【０１６１】
なお、第２の実施形態において、主要類似性が強い方向が不明である場合とは、指標HV[i_p,j]に０が設定され、かつ、指標DN[i_p,j]に０が設定された場合（縦横間で主要類似性に区別がつかず、斜めの主要類似性に区別がつかない場合）に相当する。
補間処理部１７は、縦方向の微細部類似度Cvs[i_p,j]および横方向の微細部類似度Chs[i_p,j]を算出すると、第１の実施形態と同様に、微細部類似性の強さを判定し（図１２Ｓ１３）、横方向よりも縦方向の微細部類似性が強いと判定された場合、指標HV[i_p,j]の値を１に置き換える（図１２Ｓ１４）。
【０１６２】
次に、補間処理部１７は、縦横方向の類似性を示す指標HV[i_p,j]と斜め方向の類似性を示す指標DN[i_p,j]とが如何なる値であるかを判定して（図１２Ｓ１５）、補間対象画素の類似性の強さを、以下のcase1〜case9の何れかに分類する。
case1:(HV[i_p,j],DN[i_p,j])=(1,1):縦および斜め45度方向の類似性が強い。
case2:(HV[i_p,j],DN[i_p,j])=(1,0):縦方向の類似性が強い。
【０１６３】
case3:(HV[i_p,j],DN[i_p,j])=(1,-1):縦および斜め135度方向の類似性が強い。
case4:(HV[i_p,j],DN[i_p,j])=(0,1):斜め45度方向の類似性が強い。
case5:(HV[i_p,j],DN[i_p,j])=(0,0):全ての方向の類似性が強い、または、全ての方向の類似性が弱い。
case6:(HV[i_p,j],DN[i_p,j])=(0,-1):斜め135度方向の類似性が強い。
【０１６４】
case7:(HV[i_p,j],DN[i_p,j])=(-1,1):横および斜め45度方向の類似性が強い。
case8:(HV[i_p,j],DN[i_p,j])=(-1,0):横方向の類似性が強い。
case9:(HV[i_p,j],DN[i_p,j])=(-1,-1):横および斜め135度方向の類似性が強い。
図１４は、(HV[i_p,j],DN[i_p,j])の値に対応する類似性の強い方向を示す図である。
【０１６５】
ところで、図１４では、「case5:(HV[i_p,j],DN[i_p,j])=(0,0)」に対応する表示がされていないが、case5のように、全ての方向の類似性が強い、または、全ての方向の類似性が弱い場合とは、補間対象画素が平坦部に属している、または、孤立点（周辺画素との類似性の弱い空間周波数の高い画像部分）であることに相当する。
【０１６６】
次に、補間処理部１７は、上述した判定結果に応じて、以下のようにして緑の補間値G[i,j],G[i+1,j]を算出する。

ただし、
GvL=(G[i,j-1]+G[i,j+1])/2
+(2・Z[i,j]-Z[i,j-2]-Z[i,j+2])/8
+(2・(2・G[i-1,j]-G[i-1,j-2]-G[i-1,j+2])
+2・G[i+2,j]-G[i+2,j-2]-G[i+2,j+2])/24 ・・・式77
Gv45L=(G[i,j-1]+G[i,j+1])/2
+(2・Z[i,j]-Z[i,j-2]-Z[i,j+2])/8
+(2・(2・Z[i-1,j+1]-Z[i-1,j-1]-Z[i-1,j+3])
+2・Z[i+2,j-1]-Z[i+2,j-3]-Z[i+2,j+1])/24 ・・・式78
Gv135L=(G[i,j-1]+G[i,j+1])/2
+(2・Z[i,j]-Z[i,j-2]-Z[i,j+2])/8
+(2・(2・Z[i-1,j-1]-Z[i-1,j-3]-Z[i-1,j+1])
+2・Z[i+2,j+1]-Z[i+2,j-1]-Z[i+2,j+3])/24 ・・・式79
GhL=(2・G[i-1,j]+G[i+2,j])/3
+(5・Z[i,j]-2・Z[i-3,j]-2・Z[i+1,j]-Z[i+4,j])/18
+(5・G[i,j-1]-2・G[i-3,j-1]-2・G[i+1,j-1]-G[i+4,j-1]
+5・G[i,j+1]-2・G[i-3,j+1]-2・G[i+1,j+1]-G[i+4,j+1])/36 ・・・式80
Gh45L=(2・G[i-1,j]+G[i+2,j])/3
+(5・Z[i,j]-2・Z[i-3,j]-2・Z[i+1,j]-Z[i+4,j])/18
+(5・Z[i+2,j-1]-2・Z[i-1,j-1]-2・Z[i+3,j-1]-Z[i+6,j-1]
+5・Z[i-2,j+1]-2・Z[i-1,j+1]-2・Z[i-5,j+1]-Z[i+2,j+1])/36 ・・・式81
Gh135L=(2・G[i-1,j]+G[i+2,j])/3
+(5・Z[i,j]-2・Z[i-3,j]-2・Z[i+1,j]-Z[i+4,j])/18
+(5・Z[i-2,j-1]-2・Z[i-5,j-1]-2・Z[i-1,j-1]-Z[i+2,j-1]
+5・Z[i+2,j+1]-2・Z[i-1,j+1]-2・Z[i+3,j+1]-Z[i+6,j+1])/36 ・・・式82
GvR=(G[i+1,j-1]+G[i+1,j+1])/2
+(2・Z[i+1,j]-Z[i+1,j-2]-Z[i+1,j+2])/8
+(2・G[i-1,j]-G[i-1,j-2]-G[i-1,j+2]
+2・(2・G[i+2,j]-G[i+2,j-2]-G[i+2,j+2]))/24 ・・・式83
Gv45R=(G[i+1,j-1]+G[i+1,j+1])/2
+(2・Z[i+1,j]-Z[i+1,j-2]-Z[i+1,j+2])/8
+(2・Z[i-1,j+1]-Z[i-1,j-1]-Z[i-1,j+3]
+2・(2・Z[i+2,j-1]-Z[i+2,j-3]-Z[i+2,j+1]))/24 ・・・式84
Gv135R=(G[i+1,j-1]+G[i+1,j+1])/2
+(2・Z[i+1,j]-Z[i+1,j-2]-Z[i+1,j+2])/8
+(2・Z[i-1,j-1]-Z[i-1,j-3]-Z[i-1,j+1]
+2・(2・Z[i+2,j+1]-Z[i+2,j-1]-Z[i+2,j+3]))/24 ・・・式85
GhR=(G[i-1,j]+2・G[i+2,j])/3
+(5・Z[i+1,j]-2・Z[i+4,j]-2・Z[i,j]-Z[i-3,j])/18
+(5・G[i+1,j-1]-2・G[i+4,j-1]-2・G[i,j-1]-G[i-3,j-1]
+5・G[i+1,j+1]-2・G[i+4,j+1]-2・G[i,j+1]-G[i-3,j+1])/36 ・・・式86
Gh45R=(G[i-1,j]+2・G[i+2,j])/3
+(5・Z[i+1,j]-2・Z[i+4,j]-2・Z[i,j]-Z[i-3,j])/18
+(5・Z[i+3,j-1]-2・Z[i+6,j-1]-2・Z[i+2,j-1]-Z[i-1,j-1]
+5・Z[i-1,j+1]-2・Z[i+2,j+1]-2・Z[i-2,j+1]-Z[i-5,j+1])/36 ・・・式87
Gh135R=(G[i-1,j]+2・G[i+2,j])/3
+(5・Z[i+1,j]-2・Z[i+4,j]-2・Z[i,j]-Z[i-3,j])/18
+(5・Z[i-1,j-1]-2・Z[i+2,j-1]-2・Z[i-2,j-1]-Z[i-5,j-1]
+5・Z[i+3,j+1]-2・Z[i+6,j+1]-2・Z[i+2,j+1]-Z[i-1,j+1])/36 ・・・式88
である。
【０１６７】
図１５〜図１７は、緑の補間値G[i,j],G[i+1,j]を算出する際に用いる色情報の位置を示す図である。
ところで、式77〜式88において、第１項は「緑色成分の局所的な平均情報」であり、第２項は「補間対象サブピクセルと同一の色成分による局所的な変曲情報」であり、第３項は「補間対象サブピクセルとは異なる色成分による局所的な変曲情報」である。
【０１６８】
ここで、Gv45L,Gv45Rの「補間対象画素と同一の色成分による局所的な変曲情報」は、縦方向に方向性を有する局所的な変曲情報であり、「補間対象画素と異なる色成分による局所的な変曲情報」は、縦方向および斜め45度方向に方向性を有する局所的な変曲情報であり、Gv135L,Gv135Rの「補間対象画素と同一の色成分による局所的な変曲情報」は、縦方向に方向性を有する局所的な変曲情報であり、「補間対象画素と異なる色成分による局所的な変曲情報」は、縦方向および斜め135度方向に方向性を有する局所的な変曲情報であり、Gh45L,Gh45Rの「補間対象画素と同一の色成分による局所的な変曲情報」は、横方向に方向性を有する局所的な変曲情報であり、「補間対象画素と異なる色成分による局所的な変曲情報」は、横方向および斜め45度方向に方向性を有する局所的な変曲情報であり、Gh135L,Gh135Rの「補間対象画素と同一の色成分による局所的な変曲情報」は、横方向に方向性を有する局所的な変曲情報であり、「補間対象画素と異なる色成分による局所的な変曲情報」は、横方向および斜め135度方向に方向性を有する局所的な変曲情報である。
【０１６９】
また、GvL,GvRの「補間対象画素と同一の色成分による局所的な変曲情報」と「補間対象画素と異なる色成分による局所的な変曲情報」とは、共に縦方向に方向性を有する局所的な変曲情報であり、GhL,GhRの「補間対象画素と同一の色成分による局所的な変曲情報」と「補間対象画素と異なる色成分による局所的な変曲情報」とは、共に横方向に方向性を有する局所的な変曲情報である。
【０１７０】
以上説明したように、補間処理部１７は、複数の色成分の変曲情報を用いて補間値を算出するので、第２の実施形態では、撮影光学系１２に倍率色収差が存在する場合であっても、平均情報を適切に補正することが可能であり、偽色の発生を抑制することができる。その様子を図１８に示す。
また、第２の実施形態では、複数の色成分の変曲情報が互いに相殺し合うので、色差が一定でない色境界部分における偽色も抑制できる。
【０１７１】
すなわち、第２の実施形態のＧ補間処理では、第１の実施形態のＧ補間処理によって発生する可能性が高かった偽色を抑制することができる。
ところで、複数の色成分の変曲情報は、如何なる方向に位置する色情報を採用するかで、値が変化し易い。そのため、複数の色成分の変曲情報を用いて補間値を算出する際には、補間対象画素における類似性の強い方向を正確に判定し、その方向に位置する色情報を採用した変曲情報を用いる必要がある。第２の実施形態では、縦方向や横方向の類似度成分の他に斜め方向の類似度成分を用いることによって、補間対象画素における類似性の強弱の判定の精度を向上させている。
【０１７２】
《第３の実施形態》
以下、第３の実施形態の動作を説明する。なお、第３の実施形態におけるＲＢ補間処理は、第１の実施形態と同様にして行えるため、ここでは、説明を省略する。
図１９〜図２１は、第３の実施形態における補間処理部１７の動作フローチャートであり、Ｇ補間処理における補間処理部１７の動作を示す。
【０１７３】
なお、第３の実施形態と第１の実施形態とのＧ補間処理の相違点は、縦横方向の類似性を判定する処理や指標HV[i_p,j]を設定する処理が異なる点にあり、第３の実施形態は、第１の実施形態の図２Ｓ１〜Ｓ９の処理に代えて、図１９Ｓ１〜Ｓ４、図２０、図２１の処理を行うことに相当する。
また、第３の実施形態では、指標HV[i_p,j]を設定する過程において、補間対象画素[i_p,j]の縦方向および横方向に対する主要類似度と微細部類似度とが、それぞれ異色間類似度と同色間類似度とに分けて算出される。そこで、ここでは、後述するＧ補間処理の説明を簡単にするため、補間処理部１７によって行われる各々の類似度を算出する処理を説明する。
【０１７４】
まず、縦方向および横方向に対する主要類似度の異色間類似度（以下、「主要異色間類似度」と称する。）と同色間類似度（以下、「主要同色間類似度」と称する。）とを算出する処理を説明する。
補間処理部１７は、第１の実施形態の式10〜式19によって定義される縦方向および横方向に対する複数種類の主要類似度成分（Cv1[i_p,j],Ch1[i_p,j],Cv2[i_p,j],Ch2[i_p,j],Cv3[i_p,j],Ch3[i_p,j],Cv4[i_p,j],Ch4[i_p,j],Cv5[i_p,j],Ch5[i_p,j]）を算出する。
【０１７５】
次に、補間処理部１７は、以下の式89および式90に示すようにして、加重係数a1,a2,a3によって、複数種類の同色間類似度成分を方向別に加重加算すると共に、式91および式92に示すようにして、加重係数a4,a5によって、複数種類の異色間類似度成分についても方向別に加重加算する。
CvD0[i_p,j]=(a1・Cv1[i_p,j]+a2・Cv2[i_p,j]+a3・Cv3[i_p,j])
/(a1+a2+a3) ・・・式89
ChD0[i_p,j]=(a1・Ch1[i_p,j]+a2・Ch2[i_p,j]+a3・Ch3[i_p,j])
/(a1+a2+a3) ・・・式90
CvN0[i_p,j]=(a4・Cv4[i_p,j]+a5・Cv5[i_p,j])/(a4+a5) ・・・式91
ChN0[i_p,j]=(a4・Ch4[i_p,j]+a5・Ch5[i_p,j])/(a4+a5) ・・・式92
このようにして算出されるCvD0[i_p,j]、ChD0[i_p,j]、CvN0[i_p,j]、ChN0[i_p,j]は、そのまま、補間対象画素における縦方向の主要同色間類似度CvD[i_p,j]、横方向の主要同色間類似度ChD[i_p,j]、縦方向の主要異色間類似度CvN[i_p,j]、横方向の主要異色間類似度ChN[i_p,j]とすることができるが、ここでは、周辺加算によって、これらの類似度を算出する例を示す。
【０１７６】
すなわち、補間処理部１７は、補間対象画素と周辺画素とにおける各々の類似度成分の加重加算によって得られる値（CvD0[i_p,j]、CvD0[i_p-1,j-1]、CvD0[i_p-1,j+1]、CvD0[i_p+1,j-1]、CvD0[i_p+1,j+1]など）を、類似性の連続性を考慮し、以下の《周辺加算方法１》または《周辺加算方法２》のように加重加算して、補間対象画素における縦方向の主要同色間類似度CvD[i_p,j]、横方向の主要同色間類似度ChD[i_p,j]、縦方向の主要異色間類似度CvN[i_p,j]、横方向の主要異色間類似度ChN[i_p,j]を算出する（図４に示すように補間対象画素と周辺画素とにおける類似度成分を加重加算することに相当する）。
【０１７７】
《周辺加算方法１》
CvD[i_p,j]=(4・CvD0[i_p,j]+CvD0[i_p-1,j-1]+CvD0[i_p-1,j+1]
+CvD0[i_p+1,j-1]+CvD0[i_p+1,j+1])/8 ・・・式93
ChD[i_p,j]=(4・ChD0[i_p,j]+ChD0[i_p-1,j-1]+ChD0[i_p-1,j+1]
+ChD0[i_p+1,j-1]+ChD0[i_p+1,j+1])/8 ・・・式94
CvN[i_p,j]=(4・CvN0[i_p,j]+CvN0[i_p-1,j-1]+CvN0[i_p-1,j+1]
+CvN0[i_p+1,j-1]+CvN0[i_p+1,j+1])/8 ・・・式95
ChN[i_p,j]=(4・ChN0[i_p,j]+ChN0[i_p-1,j-1]+ChN0[i_p-1,j+1]
+ChN0[i_p+1,j-1]+ChN0[i_p+1,j+1])/8 ・・・式96
《周辺加算方法２》
CvD[i_p,j]=(4・CvD0[i_p,j]
+2・(CvD0[i_p-1,j-1]+CvD0[i_p+1,j-1]+CvD0[i_p-1,j+1]+CvD0[i_p+1,j+1])
+CvD0[i_p,j-2]+CvD0[i_p,j+2]+CvD0[i_p-2,j]+CvD0[i_p+2,j])/16 ・・・式97
ChD[i_p,j]=(4・ChD0[i_p,j]
+2・(ChD0[i_p-1,j-1]+ChD0[i_p+1,j-1]+ChD0[i_p-1,j+1]+ChD0[i_p+1,j+1])
+ChD0[i_p,j-2]+ChD0[i_p,j+2]+ChD0[i_p-2,j]+ChD0[i_p+2,j])/16 ・・・式98
CvN[i_p,j]=(4・CvN0[i_p,j]
+2・(CvN0[i_p-1,j-1]+CvN0[i_p+1,j-1]+CvN0[i_p-1,j+1]+CvN0[i_p+1,j+1])
+CvN0[i_p,j-2]+CvN0[i_p,j+2]+CvN0[i_p-2,j]+CvN0[i_p+2,j])/16 ・・・式99
ChN[i_p,j]=(4・ChN0[i_p,j]
+2・(ChN0[i_p-1,j-1]+ChN0[i_p+1,j-1]+ChN0[i_p-1,j+1]+ChN0[i_p+1,j+1])
+ChN0[i_p,j-2]+ChN0[i_p,j+2]+ChN0[i_p-2,j]+ChN0[i_p+2,j])/16 ・・・式100
次に、縦方向および横方向に対する微細部類似度の異色間類似度（以下、「微細部異色間類似度」と称する。）と同色間類似度（以下、「微細部同色間類似度」と称する。）とを算出する処理を説明する。
【０１７８】
補間処理部１７は、第１の実施形態の式29〜式34によって定義される縦方向および横方向に対する複数種類の微細部類似度成分（Cvs1[i_p,j],Chs1[i_p,j],Cvs2[i_p,j],Chs2[i_p,j],Cvs3[i_p,j],Chs3[i_p,j]）を算出する。
次に、補間処理部１７は、以下の式101および式102に示すようにして、加重係数s1,s2によって、複数種類の同色間類似度成分を方向別に加重加算すると共に、式103および式104に示すようにして、異色間類似度成分をCvsN0[i_p,j]やChsNO[i_p,j]に代入する。また、縦方向の類似度成分については、以下の式101および式103によって、サブピクセル間の距離による規格化も行う。
【０１７９】
CvsD0[i_p,j]=(s1・Cvs1[i_p,j]/4+s2・Cvs2[i_p,j]/4)/(s1+s2) ・・・式101
ChsD0[i_p,j]=(s1・Chs1[i_p,j]+s2・Chs2[i_p,j])/(s1+s2) ・・・式102
CvsN0[i_p,j]=Cvs3[i_p,j]/2 ・・・式103
ChsN0[i_p,j]=Chs3[i_p,j] ・・・式104
このようにして算出されるCvsD0[i_p,j]、ChsD0[i_p,j]、CvsN0[i_p,j]、ChsN0[i_p,j]は、そのまま、補間対象画素における縦方向の微細部同色間類似度CvsD[i_p,j]、横方向の微細部同色間類似度ChsD[i_p,j]、縦方向の微細部異色間類似度CvsN[i_p,j]、横方向の微細部異色間類似度ChsN[i_p,j]とすることができるが、ここでは、周辺加算によって、これらの類似度を算出する例を示す。
【０１８０】
すなわち、補間処理部１７は、補間対象画素と周辺画素とにおける各々の類似度成分の加重加算等によって得られる値（CvsD0[i_p,j]、CvsD0[i_p-1,j-1]、CvsD0[i_p-1,j+1]、CvsD0[i_p+1,j-1]、CvsD0[i_p+1,j+1]など）を、類似性の連続性を考慮し、以下の《周辺加算方法１》または《周辺加算方法２》のように加重加算して、補間対象画素における縦方向の微細部同色間類似度CvsD[i_p,j]、横方向の微細部同色間類似度ChsD[i_p,j]、縦方向の微細部異色間類似度CvsN[i_p,j]、横方向の微細部異色間類似度ChsN[i_p,j]を算出する（図４に示すように補間対象画素と周辺画素とにおける類似度成分を加重加算することに相当する）。
【０１８１】
《周辺加算方法１》
CvsD[i_p,j]=(4・CvsD0[i_p,j]+CvsD0[i_p-1,j-1]+CvsD0[i_p-1,j+1]
+CvsD0[i_p+1,j-1]+CvsD0[i_p+1,j+1])/8 ・・・式105
ChsD[i_p,j]=(4・ChsD0[i_p,j]+ChsD0[i_p-1,j-1]+ChsD0[i_p-1,j+1]
+ChsD0[i_p+1,j-1]+ChsD0[i_p+1,j+1])/8 ・・・式106
CvsN[i_p,j]=(4・CvsN0[i_p,j]+CvsN0[i_p-1,j-1]+CvsN0[i_p-1,j+1]
+CvsN0[i_p+1,j-1]+CvsN0[i_p+1,j+1])/8 ・・・式107
ChsN[i_p,j]=(4・ChsN0[i_p,j]+ChsN0[i_p-1,j-1]+ChsN0[i_p-1,j+1]
+ChsN0[i_p+1,j-1]+ChsN0[i_p+1,j+1])/8 ・・・式108
《周辺加算方法２》
CvsD[i_p,j]=(4・CvsD0[i_p,j]+2・(CvsD0[i_p-1,j-1]+CvsD0[i_p+1,j-1]
+CvsD0[i_p-1,j+1]+CvsD0[i_p+1,j+1])+CvsD0[i_p,j-2]+CvsD0[i_p,j+2]
+CvsD0[i_p-2,j]+CvsD0[i_p+2,j])/16 ・・・式109
ChsD[i_p,j]=(4・ChsD0[i_p,j]+2・(ChsD0[i_p-1,j-1]+ChsD0[i_p+1,j-1]
+ChsD0[i_p-1,j+1]+ChsD0[i_p+1,j+1])+ChsD0[i_p,j-2]+ChsD0[i_p,j+2]
+ChsD0[i_p-2,j]+ChsD0[i_p+2,j])/16 ・・・式110
CvsN[i_p,j]=(4・CvsN0[i_p,j]+2・(CvsN0[i_p-1,j-1]+CvsN0[i_p+1,j-1]
+CvsN0[i_p-1,j+1]+CvsN0[i_p+1,j+1])+CvsN0[i_p,j-2]+CvsN0[i_p,j+2]
+CvsN0[i_p-2,j]+CvsN0[i_p+2,j])/16 ・・・式111
ChsN[i_p,j]=(4・ChsN0[i_p,j]+2・(ChsN0[i_p-1,j-1]+ChsN0[i_p+1,j-1]
+ChsN0[i_p-1,j+1]+ChsN0[i_p+1,j+1])+ChsN0[i_p,j-2]+ChsN0[i_p,j+2]
+ChsN0[i_p-2,j]+ChsN0[i_p+2,j])/16 ・・・式112
ところで、以上説明したように算出される主要異色間類似度や微細部異色間類似度は、縦方向または横方向に隣接する画素の色情報を比較することによって算出されるので、主要同色間類似度や微細部同色間類似度よりも短い距離間隔で類似性の強弱の判定が可能である。すなわち、異色間類似度には、同色間類似度よりも細かい画像の構造が反映されることになる。
【０１８２】
とりわけ、異色間類似度は、異なる色成分の色情報が全て同一の輝度情報を表していると仮定して算出されるため、異色間類似度を用いた類似性の強弱の判定は、無彩色部において、信頼性が高い。一方、同色間類似度を用いた類似性の強弱の判定は、彩色部・無彩色部ともに全般的に信頼性が高いが、画像の構造が細かい部分では、異色間類似度を用いた場合に比べて信頼性が劣る。
【０１８３】
したがって、補間処理の対象となる画像全体に対して、信頼性の高い類似性の判定を行うには、画像全体を、無彩色部と彩色部とに分け、各々の部分に適した類似度を用いる方法が優れている。第３の実施形態におけるＧ補間処理では、このような信頼性の高い類似性の判定を実現するために、補間対象画素の近傍の画像が無彩色部であるか否かによって、指標HV[i_p,j]を設定する際に用いる類似度を切り換えている。
【０１８４】
以下、第３の実施形態におけるＧ補間処理について説明するが、第１の実施形態と同じ動作については、詳細な説明を省略する。
まず、補間処理部１７は、補間対象画素が属する局所領域における色彩の有無を示す色指標を算出し（図１９Ｓ１）、色指標に基づき、補間対象画素の近傍の画像が無彩色部であるか否かを判定する（図１９Ｓ２）。
【０１８５】
なお、色指標としては、局所的な色差情報を用いることができるが、上述したように算出される主要異色間類似度や微細部異色間類似度には、類似性の強弱と同時に局所的な色差情報が反映されているので、色指標として、主要異色間類似度や微細部異色間類似度を直接利用することが可能である。
第３の実施形態では、色指標として、縦方向および横方向に対する主要異色間類似度と微細部異色間類似度とのどちらでも用いることができるが、ここでは、主要異色間類似度を用いた場合の例を示す。
【０１８６】
ただし、主要異色間類似度は、値が小さい程、類似性が強いことを示すので、縦方向および横方向に対する主要異色間類似度が共に大きな値である場合には、無彩色部で縦横両方向に対する類似性が弱いか、または、補間対象画素の近傍の画像が彩色部であることを意味する。逆に、縦方向と横方向との少なくとも１つの方向に対する主要異色間類似度が比較的小さな値であれば、補間対象画素の近傍の画像が無彩色部であり、類似性の強い方向が存在していることを意味する。
【０１８７】
すなわち、補間処理部１７は、縦方向の主要異色間類似度CvN[i_p,j]と横方向の主要異色間類似度ChN[i_p,j]とを算出し、閾値ThNv,ThNhについて、以下の条件１が成り立つか否かを判定し、条件１が成り立つ場合、補間対象画素の近傍の画像が無彩色部であると判断し、条件１が成り立たない場合、補間対象画素の近傍の画像が彩色部であると判断する。
【０１８８】
CvN[i_p,j]≦ThNv または ChN[i_p,j]≦ThNh ・・・条件１
ただし、閾値ThNv,ThNhは、階調数が256のとき10程度以下の値をとるものとする。
そして、補間処理部１７は、補間対象画素の近傍の画像が無彩色部である場合、後述する《異色間類似度を用いた指標HV[i_p,j]の設定処理》を行い（図１９Ｓ３）、補間対象画素の近傍の画像が彩色部である場合、後述する《同色間類似度を用いた指標HV[i_p,j]の設定処理》を行う（図１９Ｓ４）。
【０１８９】
このような処理によって、指標HV[i_p,j]が設定されると、補間処理部１７は、第１の実施形態と同様（図２Ｓ１０〜Ｓ１３に相当する）に、指標HV[i_p,j]の値に応じて、２つのサブピクセルの緑の補間値G[i,j]、G[i+1,j]を算出する（図１９Ｓ５〜Ｓ８）。
ここで、《異色間類似度を用いた指標HV[i_p,j]の設定処理》と《同色間類似度を用いた指標HV[i_p,j]の設定処理》とを説明する。
【０１９０】
《異色間類似度を用いた指標HV[i_p,j]の設定処理》
まず、補間処理部１７は、縦方向の主要異色間類似度CvN[i_p,j]と横方向の主要異色間類似度ChN[i_p,j]とを算出し（図２０Ｓ１）、これらの類似度に基づく類似性（以下、「主要異色間類似性」と称する。）が強い方向が不明であるか否かを判定する（図２０Ｓ２）。
【０１９１】
例えば、このような判定は、閾値Th0について、以下の条件２が成り立つか否かを判定することによって実現できる。
|CvN[i_p,j]-ChN[i_p,j]|≦Th0 ・・・条件２
そして、補間処理部１７は、このような判定結果として、主要異色間類似性が強い方向が不明である場合、縦方向の微細部異色間類似度CvsN[i_p,j]と横方向の微細部異色間類似度ChsN[i_p,j]とを算出し（図２０Ｓ３）、これらの類似度に基づく類似性（以下、「微細部異色間類似性」と称する。）が強い方向が不明であるか否かを判定する（図２０Ｓ４）。
【０１９２】
例えば、このような判定は、閾値Ths0について、以下の条件３が成り立つか否かを判定することによって実現できる。
|CvsN[i_p,j]-ChsN[i_p,j]|≦Ths0 ・・・条件３
そして、補間処理部１７は、このような判定結果として、微細部異色間類似性が強い方向が不明である場合（条件２および条件３が成り立つ場合：図２２の領域１(1)に相当する）、補間対象画素の近傍の画像が無彩色部であり、縦横間で類似性に区別がつかないと判定して指標HV[i_p,j]に０を設定する（図２０Ｓ５）。一方、微細部異色間類似性が強い方向が判別できる場合、縦方向の微細部異色間類似性が強いか否かを判定する（図２０Ｓ６）。
【０１９３】
例えば、このような判定は、以下の条件４が成り立つか否かを判定することによって実現できる。
CvsN[i_p,j]<ChsN[i_p,j] ・・・条件４
そして、補間処理部１７は、このような判定結果として、縦方向の微細部異色間類似性が強い場合（条件２および条件４が成り立ち、条件３が成り立たない場合：図２２の領域２(1)に相当する）、補間対象画素の近傍の画像が無彩色部であり、縦方向に類似性が強いと判断し、指標HV[i_p,j]に１を設定する（図２０Ｓ７）。一方、縦方向の微細部異色間類似性が強くない場合（条件２が成り立ち条件３および条件４が成り立たない場合：図２２の領域１(2)に相当する）、指標HV[i_p,j]に０を設定する（図２０Ｓ８）。
【０１９４】
また、補間処理部１７は、図２０Ｓ２の判定結果として、主要異色間類似性が強い方向が判別できる場合、縦方向の主要異色間類似性が強いか否かを判定する（図２０Ｓ９）。
例えば、このような判定は、以下の条件５が成り立つか否かを判定することによって実現できる。
【０１９５】
CvN[i_p,j]<ChN[i_p,j] ・・・条件５
そして、補間処理部１７は、このような判定結果として、縦方向の主要異色間類似性が強い場合（条件２が成り立たず条件５が成り立つ場合：図２２の領域２(2)に相当する）、指標HV[i_p,j]に１を設定する（図２０Ｓ１０）。一方、縦方向の主要異色間類似性が強くない場合（条件２および条件５が成り立たない場合：図２２の領域３に相当する）、補間対象画素の近傍の画像が無彩色部であり、横方向に類似性が強いと判断し、指標HV[i_p,j]に−１を設定する（図２０Ｓ１１）。
【０１９６】
《同色間類似度を用いた指標HV[i_p,j]の設定処理》
まず、補間処理部１７は、縦方向の主要同色間類似度CvD[i_p,j]と横方向の主要同色間類似度ChD[i_p,j]とを算出し（図２１Ｓ１）、これらの類似度に基づく類似性（以下、「主要同色間類似性」と称する。）が強い方向が不明であるか否かを判定する（図２１Ｓ２）。
【０１９７】
例えば、このような判定は、閾値Th1について、以下の条件６が成り立つか否かを判定することによって実現できる。
|CvD[i_p,j]-ChD[i_p,j]|≦Th1 ・・・条件６
そして、補間処理部１７は、このような判定結果として、主要同色間類似性が強い方向が不明である場合、縦方向の微細部同色間類似度CvsD[i_p,j]と横方向の微細部同色間類似度ChsD[i_p,j]とを算出し（図２１Ｓ３）、これらの類似度に基づく類似性（以下、「微細部同色間類似性」と称する。）が強い方向が不明であるか否かを判定する（図２１Ｓ４）。
【０１９８】
例えば、このような判定は、閾値Ths1について、以下の条件７がが成り立つか否かを判定することによって実現できる。
|CvsD[i_p,j]-ChsD[i_p,j]|≦Ths1 ・・・条件７
そして、補間処理部１７は、このような判定結果として、微細部同色間類似性が強い方向が不明である場合（条件６および条件７が成り立つ場合：図２２の領域４(1)に相当する）、補間対象画素の近傍の画像が彩色部であり、縦横間で類似性に区別がつかないと判定して指標HV[i_p,j]に０を設定する（図２１Ｓ５）。一方、微細部同色間類似性が強い方向が判別できる場合、縦方向の微細部同色間類似性が強いか否かを判定する（図２１Ｓ６）。
【０１９９】
例えば、このような判定は、以下の条件８が成り立つか否かを判定することによって実現できる。
CvsD[i_p,j]<ChsD[i_p,j] ・・・条件８
そして、補間処理部１７は、このような判定結果として、縦方向の微細部同色間類似性が強い場合（条件６および条件８が成り立ち、条件７が成り立たない場合：図２２の領域５(1)に相当する）、補間対象画素の近傍の画像が彩色部であり、縦方向に類似性が強いと判断し、指標HV[i_p,j]に１を設定する（図２１Ｓ７）。一方、縦方向の微細部同色間類似性が強くない場合（条件６が成り立ち条件７および条件８が成り立たない場合：図２２の領域４(2)に相当する）、指標HV[i_p,j]に０を設定する（図２１Ｓ８）。
【０２００】
また、補間処理部１７は、図２１Ｓ２の判定結果として、主要同色間類似性が強い方向が判別できる場合、縦方向の主要同色間類似性が強いか否かを判定する（図２１Ｓ９）。
例えば、このような判定は、以下の条件９が成り立つか否かを判定することによって実現できる。
【０２０１】
CvD[i_p,j]<ChD[i_p,j] ・・・条件９
そして、補間処理部１７は、このような判定結果として、縦方向の主要同色間類似性が強い場合（条件６が成り立たず条件９が成り立つ場合：図２２の領域５(2)に相当する）、指標HV[i_p,j]に１を設定する（図２１Ｓ１０）。一方、縦方向の主要同色間類似性が強くない場合（条件６および条件９が成り立たない場合：図２２の領域６に相当する）、補間対象画素の近傍の画像が彩色部であり、横方向に類似性が強いと判断し、指標HV[i_p,j]に−１を設定する（図２１Ｓ１１）。
【０２０２】
ところで、《異色間類似度を用いた指標HV[i_p,j]の設定処理》と《同色間類似度を用いた指標HV[i_p,j]の設定処理》とにおいて、閾値Th0、Ths0、Th1、Ths1は、縦方向の類似度と横方向の類似度との差異が微少である場合、ノイズの影響によって一方の類似性が強いと誤判定されることを避ける役割を果たす。そのため、ノイズの多いカラー画像に対しては、閾値Th0、Ths0、Th1、Ths1の値を高く設定することによって、類似性の判定の精度が高められる。
【０２０３】
以上説明したように、第３の実施形態では、局所的な色差情報が反映される異色間類似度を利用して画像全体を無彩色部と彩色部とに分け、各々の部分に適した類似度に基づいて類似性の強弱を判定することができるため、従来の技術に比べて補間量を高い精度で算出することができる。
なお、第３の実施形態では、類似性の強弱を判定する際に、異色間類似度と同色間類似度とを切り換えているが、類似性の判定に用いる類似度としては、異色間類似度と同色間類似度とを完全に切り換える以外に、無彩色部では異色間類似度の加算比率を上げ、彩色部では同色間類似度の加算比率を上げて、異色間類似度と同色間類似度とを加重加算して得られる類似度でも良い。
【０２０４】
また、第３の実施形態では、局所的な色彩の有無を調べる方法として異色間類似度に含まれる色差を用いたが、色の比など他の色指標を用いても良い。
ところで、第３の実施形態で設定される指標HV[i_p,j]は、第２の実施形態において、補間対象画素の類似性の強さを分類する際（図１３Ｓ１５に相当する）に用いることができる。すなわち、第２の実施形態の図１２Ｓ１〜Ｓ４、Ｓ１１〜Ｓ１４の処理に代えて、第３の実施形態の図１９Ｓ１〜Ｓ４、図２０、図２１の処理を行うことも可能である。
【０２０５】
《第４の実施形態》
以下、第４の実施形態の動作を説明するが、第４の実施形態におけるＧ補間処理の大まかな流れ（類似度を算出して指標を設定し、指標に基づいて補間値を算出する流れ）は、第１の実施形態と同様であるため、図示を省略する。また、第４の実施形態におけるＲＢ補間処理は、第１の実施形態と同様にして行えるため、説明を省略する。
【０２０６】
なお、第４の実施形態と第１の実施形態とのＧ補間処理の相違点は、第１の実施形態が、補間対象画素毎に縦横方向の類似性の判定や、指標HV[i_p,j]の設定を行っているのに対し、第４の実施形態は、補間対象サブピクセル毎に、縦横方向の類似性の判定と、指標HV[i,j]、HV[i+1,j]の設定を行う点にある。
すなわち、第４の実施形態では、第１の実施形態で算出される補間対象画素における縦方向の主要類似度Cv[i_p,j]に代えて、２つの補間対象サブピクセルにおける縦方向の主要類似度Cv[i,j]、Cv[i+1,j]が算出されることになり、横方向の主要類似度、縦方向の微細部類似度、横方向の微細部類似度についても同様である。
【０２０７】
そこで、ここでは、第４の実施形態の動作として、まず、縦方向および横方向に対する主要類似度（Cv[i,j],Ch[i,j],Cv[i+1,j],Ch[i+1,j]）と微細部類似度（Cvs[i,j],Chs[i,j],Cvs[i+1,j],Chs[i+1,j]）とを算出する処理を説明する。
ただし、後述する演算式では、図２８（２）の座標[i+1,j]に位置するサブピクセルに対する演算式が、座標[i,j]に位置するサブピクセルに対する演算式のiにi+1を代入することによって実現できる場合、座標[i,j]に位置するサブピクセルに対する演算式のみを記載し、座標[i+1,j]に位置するサブピクセルに対する演算式の記載は省略する。
【０２０８】
まず、座標[i,j]に位置するサブピクセルに対する縦方向の主要類似度Cv[i,j]と横方向の主要類似度Ch[i,j]とを算出する処理を説明する（座標[i+1,j]に位置するサブピクセルに対する縦方向の主要類似度Cv[i+1,j]と横方向の主要類似度Ch[i+1,j]とは、以下の式113〜式131のiにi+1を代入することによって算出できる）。
【０２０９】
補間処理部１７は、以下の式113〜式124によって定義される縦方向および横方向に対する複数種類の主要類似度成分を算出する。なお、以下の式113〜式124において、主要類似度成分は、距離に関して縦方向と横方向との間で対称な位置関係で構成される。
縦方向のGG間主要類似度成分：
Cv1[i,j]=|G[i,j-1]-G[i,j+1]| ・・・式113
横方向のGG間主要類似度成分：
Ch1[i,j]=|G[i-2,j]-G[i+2,j]| ・・・式114
縦方向のBB(RR)間主要類似度成分：
Cv2[i,j]=(|Z[i-2,j-1]-Z[i-2,j+1]|+|Z[i+2,j-1]-Z[i+2,j+1]|)/2 ・・・式115
横方向のBB(RR)間主要類似度成分：
Ch2[i,j]=(|Z[i-2,j-1]-Z[i+2,j-1]|+|Z[i-2,j+1]-Z[i+2,j+1]|)/2 ・・・式116
縦方向のRR(BB)間主要類似度成分：
Cv3[i,j]=(|Z[i,j-2]-Z[i,j]|+|Z[i,j+2]-Z[i,j]|)/2 ・・・式117
横方向のRR(BB)間主要類似度成分：
Ch3[i,j]=(|Z[i-4,j]-Z[i,j]|+|Z[i+4,j]-Z[i,j]|)/2 ・・・式118
縦方向のGR(GB)間主要類似度成分：
Cv4[i,j]=(|G[i,j-1]-Z[i,j]|+|G[i,j+1]-Z[i,j]|)/2 ・・・式119
横方向のGR(GB)間主要類似度成分：
Ch4[i,j]=(|G[i-2,j]-Z[i,j]|+|G[i+2,j]-Z[i,j]|)/2 ・・・式120
縦方向のBG(RG)間主要類似度成分：
Cv5[i,j]=(|Z[i-2,j-1]-G[i-2,j]|+|Z[i-2,j+1]-G[i-2,j]|
+|Z[i+2,j-1]-G[i+2,j]|+|Z[i+2,j+1]-G[i+2,j]|)/4 ・・・式121
横方向のBG(RG)間主要類似度成分：
Ch5[i,j]=(|Z[i-2,j-1]-G[i,j-1]|+|Z[i-2,j+1]-G[i,j+1]|
+|Z[i+2,j-1]-G[i,j-1]|+|Z[i+2,j+1]-G[i,j+1]|)/4 ・・・式122
縦方向の輝度間主要類似度成分：
Cv6[i,j]=(|Y[i,j-1]-Y[i,j]|+|Y[i,j+1]-Y[i,j]|)/2 ・・・式123
横方向の輝度間主要類似度成分：
Ch6[i,j]=(|Y[i-2,j]-Y[i,j]|+|Y[i+2,j]-Y[i,j]|)/2 ・・・式124
ただし、式123および式124において、Y[i,j]は、以下の式125によって算出される値であり、補間対象サブピクセルの近傍に位置するサブピクセルの色成分の色情報をR:G:B=1:2:1の比で平均化するフィルタリング処理で生成される輝度に相当する。
【０２１０】
Y[i,j]=(4・(A[i,j]+A[i-1,j]+A[i+1,j])
+2・(A[i,j-1]+A[i-1,j-1]+A[i+1,j-1]
+A[i,j+1]+A[i-1,j+1]+A[i+1,j+1]
+A[i-2,j]+A[i+2,j])
+A[i-2,j-1]+A[i-2,j+1]+A[i+2,j-1]+A[i+2,j+1])/32 ・・・式125
ただし、式125において、A[i,j]は、サブピクセル型ベイア配列上の任意の色情報を表し、配置場所に応じてGまたはZの値をとる。図２３は、輝度を生成するためのフィルタを示す図である。
【０２１１】
なお、上述した各々の主要類似度成分を構成する類似度の要素は、差分の絶対値を用いて算出しているが、絶対値の２乗やべき乗等によって算出しても良い。次に、補間処理部１７は、以下の式126および式127に示すようにして、第１の実施形態と同様の役割を果たす加重係数a1,a2,a3,a4,a5,a6によって、複数種類の主要類似度成分を各々の方向毎に加重加算する。
【０２１２】
Cv0[i,j]=(a1・Cv1[i,j]+a2・Cv2[i,j]+a3・Cv3[i,j]+a4・Cv4[i,j]
+a5・Cv5[i,j]+a6・Cv6[i,j])/(a1+a2+a3+a4+a5+a6) ・・・式126
Ch0[i,j]=(a1・Ch1[i,j]+a2・Ch2[i,j]+a3・Ch3[i,j]+a4・Ch4[i,j]
+a5・Ch5[i,j]+a6・Ch6[i,j])/(a1+a2+a3+a4+a5+a6) ・・・式127
このようにして算出されるCv0[i,j]およびCh0[i,j]は、そのまま、補間対象サブピクセルの縦方向の主要類似度Cv[i,j]および横方向の主要類似度Ch[i,j]とすることができるが、ここでは、周辺加算によって、これらの類似度を算出する例を示す。
【０２１３】
すなわち、補間処理部１７は、補間対象サブピクセルと、補間対象サブピクセルの近傍に位置するサブピクセル（以下、「周辺サブピクセル」と称する。）とにおける主要類似度成分の方向別の加重加算よって得られる値（Cv0[i,j]、Cv0[i-1,j-1]、Cv0[i-1,j+1]、Cv0[i+1,j-1]、Cv0[i+1,j+1]など）を、以下の《周辺加算方法１》または《周辺加算方法２》のように加重加算して、補間対象サブピクセルの縦方向の主要類似度Cv[i,j]および横方向の主要類似度Ch[i,j]を算出する。
【０２１４】
《周辺加算方法１》
Cv[i,j]=(4・Cv0[i,j]+Cv0[i-2,j-1]+Cv0[i-2,j+1]+Cv0[i+2,j-1]
+Cv0[i+2,j+1])/8 ・・・式128
Ch[i,j]=(4・Ch0[i,j]+Ch0[i-2,j-1]+Ch0[i-2,j+1]+Ch0[i+2,j-1]
+Ch0[i+2,j+1])/8 ・・・式129
《周辺加算方法２》
Cv[i,j]=(4・Cv0[i,j]
+2・(Cv0[i-2,j-1]+Cv0[i+2,j-1]+Cv0[i-2,j+1]+Cv0[i+2,j+1])
+Cv0[i,j-2]+Cv0[i,j+2]+Cv0[i-4,j]+Cv0[i+4,j])/16 ・・・式130
Ch[i,j]=(4・Ch0[i,j]
+2・(Ch0[i-2,j-1]+Ch0[i+2,j-1]+Ch0[i-2,j+1]+Ch0[i+2,j+1])
+Ch0[i,j-2]+Ch0[i,j+2]+Ch0[i-4,j]+Ch0[i+4,j])/16 ・・・式131
なお、《周辺加算方法１》は、図２４（１）に示すようにして補間対象サブピクセルと周辺サブピクセルとにおける主要類似度成分の加重加算を行うことに相当し、《周辺加算方法２》は、図２４（２）に示すようにして補間対象サブピクセルと周辺サブピクセルとにおける主要類似度成分の加重加算を行うことに相当する。
【０２１５】
次に、座標[i,j]に位置するサブピクセルに対する縦方向の微細部類似度Cvs[i,j]および横方向の微細部類似度Chs[i,j]と、座標[i+1,j]に位置するサブピクセルに対する縦方向の微細部類似度Cvs[i+1,j]および横方向の微細部類似度Chs[i+1,j]とを算出する処理を説明する。
【０２１６】
補間処理部１７は、以下の式132〜式143によって定義される縦方向および横方向に対する複数種類の微細部類似度成分を算出する。なお、以下の式132〜式143において、微細部類似度成分は、距離に関して縦方向と横方向との間で非対称な位置関係で構成される。
縦方向のRR(BB)間微細部類似度成分：
Cvs1[i,j]=(|Z[i,j-2]-Z[i,j]|+|Z[i,j+2]-Z[i,j]|)/2 ・・・式132
Cvs1[i+1,j]=(|Z[i+1,j-2]-Z[i+1,j]|+|Z[i+1,j+2]-Z[i+1,j]|)/2 ・・・式133
横方向のRR(BB)間微細部類似度成分：
Chs1[i,j]=|Z[i,j]-Z[i+1,j]| ・・・式134
Chs1[i+1,j]=|Z[i,j]-Z[i+1,j]| ・・・式135
縦方向のGG間微細部類似度成分：
Cvs2[i,j]=|G[i,j-1]-G[i,j+1]| ・・・式136
Cvs2[i+1,j]=|G[i+1,j-1]-G[i+1,j+1]| ・・・式137
横方向のGG間微細部類似度成分：
Chs2[i,j]=(|G[i,j-1]-G[i+1,j-1]|+|G[i,j+1]-G[i+1,j+1]|)/2 ・・・式138
Chs2[i+1,j]=(|G[i,j-1]-G[i+1,j-1]|+|G[i,j+1]-G[i+1,j+1]|)/2 ・・・式139
縦方向のGR(GB)間微細部類似度成分：
Cvs3[i,j]=(|G[i,j-1]-Z[i,j]|+|G[i,j+1]-Z[i,j]|)/2 ・・・式140
Cvs3[i+1,j]=(|G[i+1,j-1]-Z[i+1,j]|+|G[i+1,j+1]-Z[i+1,j]|)/2 ・・・式141
横方向のGR(GB)間微細部類似度成分：
Chs3[i,j]=|G[i-1,j]-Z[i,j]| ・・・式142
Chs3[i+1,j]=|G[i+2,j]-Z[i+1,j]| ・・・式143
なお、上述した各々の微細部類似度成分を構成する類似度の要素は、差分の絶対値を用いて算出しているが、絶対値の２乗やべき乗等によって算出しても良い。
【０２１７】
次に、補間処理部１７は、以下の式144および式145に示すようにして、加重係数s1,s2,s3によって、複数種類の微細部類似度成分を方向別に加重加算する（座標[i+1,j]に位置するサブピクセルに対する複数種類の微細部類似度成分についても、同様にして方向別の加重加算が行える）。なお、式144では、縦方向の類似度成分を、サブピクセル間の距離による規格化して用いている。
【０２１８】
Cvs0[i,j]=(s1・Cvs1[i,j]/4+s2・Cvs2[i,j]/4+s3・Cvs3[i,j]/2)
/(s1+s2+s3) ・・・式144
Chs0[i,j]=(s1・Chs1[i,j]+s2・Chs2[i,j]+s3・Chs3[i,j])
/(s1+s2+s3) ・・・式145
このようにして算出されるCvs0[i,j]およびChs0[i,j]は、そのまま、補間対象サブピクセルの縦方向の微細部類似度Cvs[i,j]および横方向の微細部類似度Chs[i,j]とすることができるが、ここでは、周辺加算によって、これらの類似度を算出する例を示す。
【０２１９】
すなわち、補間処理部１７は、補間対象サブピクセルと周辺サブピクセルとにおける微細部類似度成分の方向別の加重加算よって得られる値（Cvs0[i,j]、Cvs0[i-1,j-1]、Cvs0[i-1,j+1]、Cvs0[i+1,j-1]、Cvs0[i+1,j+1]など）を、以下の《周辺加算方法１》または《周辺加算方法２》のように加重加算して、縦方向の微細部類似度Cvs[i,j]および横方向の微細部類似度Chs[i,j]を算出する（座標[i+1,j]に位置するサブピクセルに対する微細部類似度についても、同様にして算出する）。
【０２２０】
《周辺加算方法１》
Cvs[i,j]=(4・Cvs0[i,j]+Cvs0[i-2,j-1]+Cvs0[i-2,j+1]+Cvs0[i+2,j-1]
+Cvs0[i+2,j+1])/8 ・・・式146
Chs[i,j]=(4・Chs0[i,j]+Chs0[i-2,j-1]+Chs0[i-2,j+1]+Chs0[i+2,j-1]
+Chs0[i+2,j+1])/8 ・・・式147
《周辺加算方法２》
Cvs[i,j]=(4・Cvs0[i,j]
+2・(Cvs0[i-2,j-1]+Cvs0[i+2,j-1]+Cvs0[i-2,j+1]+Cvs0[i+2,j+1])
+Cvs0[i,j-2]+Cvs0[i,j+2]+Cvs0[i-4,j]+Cvs0[i+4,j])/16 ・・・式148
Chs[i,j]=(4・Chs0[i,j]
+2・(Chs0[i-2,j-1]+Chs0[i+2,j-1]+Chs0[i-2,j+1]+Chs0[i+2,j+1])
+Chs0[i,j-2]+Chs0[i,j+2]+Chs0[i-4,j]+Chs0[i+4,j])/16 ・・・式149
ところで、第４の実施形態の２つの補間対象サブピクセルにおける主要類似性や微細部類似性の判定および指標HV[i,j]、HV[i+1,j]の設定は、上述したように算出される主要類似度や微細部類似度を用いることによって、第１の実施形態の補間対象画素における処理と同様に算出することができる。すなわち、第１の実施形態の図２Ｓ２〜Ｓ６、Ｓ８、Ｓ９の説明において、図２８（１）の座標[i_p,j]に位置する補間対象画素に代えて、図２８（２）の座標[i,j]や[i+1,j]に位置する補間対象サブピクセルに対する処理を行うことに相当する。
【０２２１】
このようにして、指標HV[i,j]やHV[i+1,j]の設定が完了すると、補間処理部１７は、以下のようにして、２つの補間対象サブピクセルの緑の補間値G[i,j]、G[i+1,j]を算出する。
HV[i,j]=1のとき（縦方向に類似性が強い） G[i,j]=GvL
HV[i+1,j]=1のとき（縦方向に類似性が強い） G[i+1,j]=GvR
HV[i,j]=0のとき（縦横方向に類似性が強い、または弱い）
G[i,j]=(GvL+GhL)/2
HV[i+1,j]=0のとき（縦横方向に類似性が強い、または弱い）
G[i+1,j]=(GvR+GhR)/2
HV[i,j]=-1のとき（横方向に類似性が強い） G[i,j]=GhL
HV[i+1,j]=-1のとき（横方向に類似性が強い） G[i+1,j]=GhR
ただし、GvL,GvR,GhL,GhRは、第１の実施形態の《平均補間》または《色差補間》のようにして算出される値である。
【０２２２】
以上説明したように、第４の実施形態では、補間対象サブピクセル毎に、類似度を算出し、その類似度に基づく類似性の判定結果に応じて補間値を算出することができる。
【０２２３】
なお、第４の実施形態では、第１の実施形態と同様にして、縦方向および横方向に対して主要類似度や微細部類似度を算出し、緑の補間値を算出しているが、例えば、式150〜式153に示すようにして、斜め45度方向の主要類似度C45[i,j]、C45[i+1,j]および斜め135度方向の主要類似度C135[i,j]、C135[i+1,j]を算出し、これらの類似度に基づき、第２の実施形態と同様に、補間対象サブピクセルの斜め主要類似性を判定して緑の補間値を算出しても良い。
【０２２４】
C45[i,j]=(|R[i+4,j-2]-R[i,j]|-|R[i-4,j+2]-R[i,j]|)/2 ・・・式150
C45[i+1,j]=(|R[i+5,j-2]-R[i+1,j]|-|R[i-3,j+2]-R[i+1,j]|)/2 ・・・式151
C135[i,j]=(|R[i-4,j-2]-R[i,j]|-|R[i+4,j+2]-R[i,j]|)/2 ・・・式152
C135[i+1,j]=(|R[i-3,j-2]-R[i+1,j]|-|R[i+5,j+2]-R[i+1,j]|)/2 ・・・式153
また、第４の実施形態では、第１の実施形態と同様にして、指標HV[i,j]、HV[i+1,j]の値を設定しているが、指標HV[i,j]やHV[i+1,j]の設定は、第３の実施形態と同様に算出しても良い。すなわち、第３の実施形態と同様に、主要類似度を主要異色間類似度と主要同色間類似度とに分け、微細部類似度を微細部異色間類似度と微細部同色間類似度とに分け、異色間類似度と同色間類似度とを切り換えて用いる処理を、図２８（１）の座標[i_p,j]に位置する補間対象画素に代えて、図２８（２）の座標[i,j]や[i+1,j]に位置する補間対象サブピクセルに対して個別に当てはめた処理として行えば良い。
【０２２５】
《第５の実施形態》
以下、第５の実施形態の動作を説明する。なお、第５の実施形態におけるＲＢ補間処理は、第１の実施形態と同様にして行えるため、ここでは、説明を省略する。
図２５は、第５の実施形態における補間処理部１７の動作フローチャートであり、Ｇ補間処理における補間処理部１７の動作を示す。ただし、図２５では、ベイア配列上の画素の左側に位置するサブピクセルを補間対象とした場合の例を示す。
【０２２６】
以下、図２５に基づいて、第５の実施形態におけるＧ補間処理について説明する。
まず、補間処理部１７は、補間対象サブピクセルにおける縦方向および横方向の類似度を算出する（図２５Ｓ１）。
ここで、座標[i,j]に位置する補間対象サブピクセルにおける縦方向の類似度Cv[i,j]および横方向の類似度Ch[i,j]を算出する処理の詳細を説明する。
【０２２７】
補間処理部１７は、式154〜式166によって定義される縦方向および横方向に対する複数種類の類似度成分を算出する。なお、式154〜式166では、上述した実施形態のような対称関係にこだわらず、近隣の画素同士による類似度成分の算出を行う。
縦方向のGG間類似度成分：
Cv1[i,j]=|G[i,j-1]-G[i,j+1]| ・・・式154
横方向のGG間類似度成分：
Gh1[i,j]=|G[i-1,j]-G[i+2,j]| ・・・式155
縦方向のBB(RR)間類似度成分：
Cv2[i,j]=(|Z[i-1,j-1]-Z[i-1,j+1]|+|Z[i+2,j-1]-Z[i+2,j+1]|)/2 ・・・式156
横方向のBB(RR)間類似度成分：
Ch2[i,j]=(|Z[i-1,j-1]-Z[i+2,j-1]|+|Z[i-1,j+1]-Z[i+2,j+1]|)/2 ・・・式157
縦方向のRR(BB)間類似度成分：
Cv3[i,j]=(|Z[i,j-2]-Z[i,j]|+|Z[i,j+2]-Z[i,j]|)/2 ・・・式158
横方向のRR(BB)間類似度成分：
Ch3[i,j]=(|Z[i-3,j]-Z[i,j]|+3・|Z[i+1,j]-Z[i,j]|)/4 ・・・式159
縦方向のGR(GB)間類似度成分：
Cv4[i,j]=(|G[i,j-1]-Z[i,j]|+|G[i,j+11-Z[i,j]|)/2 ・・・式160
横方向のGR(GB)間類似度成分：
Gh4[i,j]=(2・|G[i-1,j]-Z[i,j]|+|G[i+2,j]-Z[i,j]|)/3 ・・・式161
縦方向のBG(RG)間類似度成分：
Cv5[i,j]=(|Z[i-1,j-1]-G[i-1,j]|+|Z[i-1,j+1]-G[i-1,j]|
+|Z[i+2,j-1]-G[i+2,j]|+|Z[i+2,j+1]-G[i+2,j]|)/4 ・・・式162
横方向のBG(RG)間類似度成分：
Ch5[i,j]=(2・|Z[i-1,j-1]-G[i,j-1]|+2・|Z[i-1,j+1]-G[i,j+1]|
+|Z[i+2,j-1]-G[i,j-1]|+|Z[i+2,j+1]-G[i,j+1]|)/6 ・・・式163
縦方向の輝度間類似度成分：
Cv6[i,j]=(|Y[i,j-1]-Y[i,j]|+|Y[i,j+1]-Y[i,j]|)/2 ・・・式164
横方向の輝度間類似度成分：
Ch6[i,j]=(2・|Y[i-1,j]-Y[i,j]|+|Y[i+2,j]-Y[i,j]|)/3 ・・・式165
ただし、式164および式165において、Y[i,j]は、式166によって算出される値であり、補間対象サブピクセルの近傍に位置するサブピクセルの色成分の色情報をR:G:B=1:2:1の比で平均化するフィルタリング処理で生成される輝度に相当する。
【０２２８】
Y[i,j]=(4・A[i,j]+2・(A[i,j-1]+A[i,j+1]+A[i-1,j]+A[i+2,j])
+A[i-1,j-1]+A[i-1.j+1]+A[i+2,j-11+A[i+2,j+1])/16 ・・・式166
ただし、式166において、A[i,j]は、サブピクセル型ベイア配列上の任意の色情報を表し、配置場所に応じてGまたはZの値をとる。
なお、上述した各々の類似度成分を構成する類似度の要素は、差分の絶対値を用いて算出しているが、絶対値の２乗やべき乗等によって算出しても良い。
【０２２９】
次に、補間処理部１７は、第４の実施形態の式126および式127と同様に（第１の実施形態の式23および式24の加重係数と同様の役割を果たす加重係数を用いて）、複数種類の類似度成分を各々の方向毎に加重加算して、Cv0[i,j]およびCh0[i,j]を算出する。
このようにして算出されるCv0[i,j]およびCh0[i,j]は、そのまま、補間対象サブピクセルの縦方向の類似度Cv[i,j]および横方向の類似度Ch[i,j]とすることができるが、ここでは、周辺加算によって、これらの類似度を算出する例を示す。
【０２３０】
すなわち、補間処理部１７は、補間対象サブピクセルと、周辺サブピクセルとにおける類似度成分の方向別の加重加算よって得られる値（Cv0[i,j]、Cv0[i+1,j]、Cv0[i-1,j-1]、Cv0[i-1,j+1]、Cv0[i+2,j-1]、Cv0[i+2,j+1]など）を、以下の《周辺加算方法１》または《周辺加算方法２》のように加重加算して、補間対象サブピクセルの縦方向の類似度Cv[i,j]および横方向の類似度Ch[i,j]を算出する。
【０２３１】
《周辺加算方法１》
Cv[i,j]=(2・(Cv0[i,j]+Cv0[i+1,j])+Cv0[i-1,j-1]+Cv0[i-1,j+1]
+Cv0[i+2,j-1]+Cv0[i+2,j+1])/8 ・・・式167
Ch[i,j]=(2・(Ch0[i,j]+Ch0[i+1,j])+Ch0[i-1,j-1]+Ch0[i-1,j+1]
+Ch0[i+2,j-1]+Ch0[i+2,j+1])/8 ・・・式168
《周辺加算方法２》
Cv[i,j]=(2・(Cv0[i,j]+Cv0[i+1,j])
+2・(Cv0[i-1,j-1]+Cv0[i-1,j+1]+Cv0[i+2,j-1]+Cv0[i+2,j+1])
+(Cv0[i,j-2]+Cv0[i+1,j-2])/2
+(Cv0[i,j+2]+Cv0[i+1,j+2])/2
+Cv0[i-3,j]+Cv0[i+4,j])/16 ・・・式169
Ch[i,j]=(2・(Ch0[i,j]+Ch0[i+1,j])
+2・(Ch0[i-1,j-1]+Ch0[i-1,j+1]+Ch0[i+2,j-1]+Ch0[i+2,j+1])
+(Ch0[i,j-2]+Ch0[i+1,j-2])/2
+(Ch0[i,j+2]+Ch0[i+1,j+2])/2
+Ch0[i-3,j]+Ch0[i+4,j])/16 ・・・式170
なお、座標[i,j]に位置するサブピクセルの類似度を算出する際に用いる色情報の位置と、座標[i+1,j]に位置するサブピクセル（ベイア配列上の画素の左側に位置するサブピクセル）の類似度を算出する際に用いる色情報の位置とが、図２８（２）のサブピクセル型ベイア配列のi列目のサブピクセルとi+1列目のサブピクセルとの間を通る直線について、左右対称であるすると、座標[i+1,j]に位置するサブピクセルの縦方向の類似度Cv[i+1,j]および横方向の類似度Ch[i+1,j]は、式154〜式170に記載のi-3〜i+4を、以下のように置き換えることによって算出できる。
【０２３２】
i-3→i+4,i-2→i+3,i-1→i+2,i→i+1,i+1→i,i+2→i-1,i+3→i-2,i+4→i-3。
補間処理部１７は、以上説明したようにして縦方向および横方向の類似度を算出すると、これらの類似度に基づき、補間対象サブピクセルにおける縦方向および横方向に対する類似性（以下、「縦横類似性」と称する。）を判定する（図２５Ｓ２）。そして、補間処理部１７は、第１の実施形態で縦横主要類似性の判定結果として指標HV[i_p,j]に値を設定した処理と同様に、縦横類似性の判定結果として指標HV[i,j]に値を設定する（図２５Ｓ３〜Ｓ５）。
【０２３３】
なお、座標[i+1,j]に位置するサブピクセルの縦横類似性を示す指標HV[i+1,j]の設定は、上述したようにして算出される縦方向の類似度Cv[i+1,j]および横方向の類似度Ch[i+1,j]を用いることによって、指標HV[i,j]と同様に設定することができる。
次に、補間処理部１７は、指標HV[i,j]が如何なる値であるかを判定し（図２５Ｓ６）、以下のようにして座標[i,j]に位置する補間対象サブピクセルの緑の補間値G[i,j]を算出する。
【０２３４】
HV[i,j]=1のとき G[i,j]=GvL：図２５Ｓ７
HV[i,j]=0のとき G[i,j]=(GvL+GhL)/2：図２５Ｓ８
HV[i,j]=1のとき G[i,j]=GhL：図２５Ｓ９
ただし、GvL,GhLは、第１の実施形態の式41,式43または式45,式47のようにして算出される値である。
【０２３５】
なお、座標[i+1,j]に位置する補間対象サブピクセルの緑の補間値G[i+1,j]は、上述したように設定される指標HV[i+1,j]が如何なる値であるかを判定することによって、以下のように算出することができる。
HV[i+1,j]=1のとき G[i+1,j]=GvR
HV[i+1,j]=0のとき G[i+1,j]=(GvR+GhR)/2
HV[i+1,j]=1のとき G[i+1,j]=GhR
ただし、GvR,GhRは、第１の実施形態の式42,式44または式46,式48のようにして算出される値である。
【０２３６】
以上説明したように、第５の実施形態では、補間対象サブピクセル毎に、類似度を算出し、その類似度に基づく類似性の判定結果に応じて補間値を算出することができる。
なお、第５の実施形態では、第１の実施形態の式23および式24の加重係数と同様の役割を果たす加重係数によって、式154〜式166で定義される複数種類の類似度成分を加重加算して、補間対象サブピクセルの縦方向の類似度Cv[i,j]および横方向の類似度Ch[i,j]を算出し、このような類似度に基づいて指標HV[i,j]に値を設定しているが、指標HV[i,j]の設定は、以下に示す処理（請求項４５ないし請求項５２に対応する）で実現することも可能である。
【０２３７】
まず、補間処理部１７は、式154〜式163で定義される複数種類の類似度成分を算出し、これらの類似度成分を第３の実施形態の式89〜式92の加重係数と同様の役割を果たす加重係数によって加重加算し、補間対象サブピクセルにおける縦方向の同色間類似度CvD[i,j]と、横方向の同色間類似度ChD[i,j]と、縦方向の異色間類似度CvN[i,j]と、横方向の異色間類似度ChN[i,j]とを算出する。なお、これらの類似度の算出の過程では、式167〜式170と同様にして、周辺加算を行うものとする。
【０２３８】
次に、補間処理部１７は、縦方向の異色間類似度CvN[i,j]と横方向の異色間類似度ChN[i,j]とに対し、第３の実施形態の条件１と同様の条件が成り立つか否かを判定することによって、補間対象サブピクセルの近傍の画像が無彩色部であるか彩色部であるかを判定する。
そして、補間処理部１７は、補間対象サブピクセルの近傍の画像が無彩色部である場合、縦方向の異色間類似度CvN[i,j]と横方向の異色間類似度ChN[i,j]とに基づき、縦横類似性を判定して指標HV[i,j]に値を設定し、補間対象サブピクセルの近傍の画像が彩色部である場合、縦方向の同色間類似度CvD[i,j]と横方向の同色間類似度ChD[i,j]とに基づき、縦横類似性を判定して指標HV[i,j]に値を設定する。なお、このような縦横類似性の判定や指標HV[i,j]の設定の処理は、第１の実施形態で縦横主要類似性の判定結果として指標HV[i_p,j]に値を設定した処理と同様に行うものとする。
【０２３９】
すなわち、以上説明したようにして設定された指標HV[i,j]には、第３の実施形態の指標HV[i_p,j]と同様に、無彩色部と彩色部との各々の部分に適した類似度に基づいて判定された縦横類似性が反映されることになる。
《第６の実施形態》
以下、第６の実施形態の動作を説明する。なお、第６の実施形態におけるＲＢ補間処理は、第１の実施形態と同様にして行えるため、ここでは、説明を省略する。
【０２４０】
図２６は、第６の実施形態における補間処理部１７の動作フローチャートであり、Ｇ補間処理における補間処理部１７の動作を示す。ただし、図２６では、ベイア配列上の画素の左側に位置するサブピクセルを補間対象とした場合の例を示す。
以下、第６の実施形態におけるＧ補間処理について説明するが、第５の実施形態と同じ動作については、詳細な説明を省略する。
【０２４１】
まず、補間処理部１７は、第５の実施形態と同様に、補間対象サブピクセルにおける縦方向および横方向の類似度を算出して（図２６Ｓ１）、縦横主要類似性を比較し（図２６Ｓ２）、指標HV[i,j]に値を設定する（図２６Ｓ３〜Ｓ５）。
次に、補間処理部１７は、補間対象サブピクセルにおける斜め45度方向および斜め135度方向の類似度を算出する（図２６Ｓ６）。
【０２４２】
ここで、座標[i,j]に位置する補間対象サブピクセルにおける斜め45度方向の類似度C45[i,j]および斜め135度方向の類似度C135[i,j]を算出する処理の詳細を説明する。
補間処理部１７は、補間処理部１７は、式171〜式178によって定義される斜め45度方向および斜め135度方向に対する複数種類の類似度成分を算出する。
【０２４３】
斜め45度方向のGG間類似度成分：
C45_1[i,j]=(|G[i-2,j]-G[i,j-1]|+|G[i-1,j]-G[i+1,j-1]|
+|G[i,j+11-G[i+2,j]|+|G[i+1,j+1]-G[i+3,j]|)/4 ・・・式171
斜め135度方向のGG間類似度成分：
C135_1[i,j]=(|G[i-2,j]-G[i,j+1]|+|G[i-1,j]-G[i+1,j+1]|
+|G[i,j-1]-G[i+2,j]|+|G[i+1,j-1]-G[i+3,j]|)/4 ・・・式172
斜め45度方向のBB(RR)間類似度成分：
C45_2[i,j]=(|Z[i-2,j+1]-Z[i+2,j-1]|
+|Z[i-1,j+1]-Z[i+3,j-1]|)/2 ・・・式173
斜め135度方向のBB(RR)間類似度成分：
C135_2[i,j]=(|Z[i-2,j-1]-Z[i+2,j+1]|
+|Z[i-1,j-1]-Z[i+3,j+1]|)/2 ・・・式174
斜め45度方向のRR(BB)間類似度成分：
C45_3[i,j]=(|Z[i-4,j+2]-Z[i,j]|+|Z[i-3,j+2]-Z[i+1,j]|
+|Z[i+4,j-2]-Z[i,j]|+|Z[i+5,j-2]-Z[i+1,j]|)/4 ・・・式175
斜め135度方向のRR(BB)間類似度成分：
C135_3[i,j]=(|Z[i-4,j-2]-Z[i,j]|+|Z[i-3,j-2]-Z[i+1,j]|
+|Z[i+4,j+2]-Z[i,j]|+|Z[i+5,j+2]-Z[i+1,j]|)/4 ・・・式176
斜め45度方向のBR(RB)間類似度成分：
C45_4[i,j]=(|Z[i-2,j+1]-Z[i,j]|+|Z[i-1,j+1]-Z[i+1,j]|
+|Z[i+2,j-1]-Z[i,j]|+|Z[i+3,j-1]-Z[i+1,j]|)/4 ・・・式177
斜め135度方向のBR(RB)間類似度成分：
C135_4[i,j]=(|Z[i-2,j-1]-Z[i,j]|+|Z[i-1,j-1]-Z[i+1,j]|
+|Z[i+2,j+1]-Z[i,j]|+|Z[i+3,j+1]-Z[i+1,j]|)/4 ・・・式178
なお、上述した各々の類似度成分を構成する類似度の要素は、差分の絶対値を用いて算出しているが、絶対値の２乗やべき乗等によって算出しても良い。
【０２４４】
次に、補間処理部１７は、式179および式180に示すようにして、第２の実施形態と同様の役割を果たす加重係数b1,b2,b3,b4によって、複数種類の類似度成分を各々の方向毎に加重加算する。
C45_0[i,j]=(b1・C45_1[i,j]+b2・C45_2[i,j]+b3・C45_3[i,j]
+b4・C45_4[i,j])/(b1+b2+b3+b4) ・・・式179
C135_0[i,j]=(b1・C135_1[i,j]+b2・C135_2[i_p,j]+b3・C135_3[i,j]
+b4・C135_4[i,j])/(b1+b2+b3+b4) ・・・式180
このようにして算出されるC45_0[i,j]およびC135_0[i,j]は、そのまま、補間対象サブピクセルの斜め方向の類似度C45[i,j]、C135[i,j]とすることができるが、ここでは、周辺加算によって、これらの類似度を算出する例を示す。
【０２４５】
すなわち、補間処理部１７は、補間対象サブピクセルと、周辺サブピクセルとにおける類似度成分の方向別の加重加算よって得られる値（C45_0[i,j]、C45_0[i+1,j]、C45_0[i-1,j-1]、C45_0[i-1,j+1]、C45_0[i+2,j-1]、C45_0[i+2,j+1]など）を、以下の《周辺加算方法１》または《周辺加算方法２》のように加重加算して、補間対象サブピクセルの斜め方向の類似度C45[i,j]、C135[i,j]を算出する。
【０２４６】
《周辺加算方法１》
C45[i,j]=(2・(C45_0[i,j]+C45_0[i+1,j])+C45_0[i-1,j-1]+C45_0[i-1,j+1]
+C45_0[i+2,j-1]+C45_0[i+2,j+1])/8 ・・・式181
C135[i,j]=(2・(C135_0[i,j]+C135_0[i+1,j])+C135_0[i-1,j-1]
+C135_0[i-1,j+1]+C135_0[i+2,j-1]+C135_0[i+2,j+1])/8 ・・・式182
《周辺加算方法２》
C45[i,j]=(2・(C45_0[i,j]+C45_0[i+1,j])
+2・(C45_0[i-1,j-1]+C45_0[i-1,j+1]+C45_0[i+2,j-1]+C45_0[i+2,j+1])
+(C45_0[i,j-2]+C45_0[i+1,j-2])/2
+(C45_0[i,j+2]+C45_0[i+1,j+2])/2
+C45_0[i-3,j]+C45_0[i+4,j])/16 ・・・式183
C135[i,j]=(2・(C135_0[i,j]+C135_0[i+1,j])
+2・(C135_0[i-1,j-1]+C135_0[i-1,j+1]+C135_0[i+2,j-1]+C135_0[i+2,j+1]
+(C135_0[i,j-2]+C135_0[i+1,j-2])/2
+(C135_0[i,j+2]+C135_0[i+1,j+2])/2
+C135_0[i-3,j]+C135_0[i+4,j])/16 ・・・式184
なお、座標[i,j]に位置するサブピクセルの類似度を算出する際に用いる色情報の位置と、座標[i+1,j]に位置するサブピクセル（ベイア配列上の画素の左側に位置するサブピクセル）の類似度を算出する際に用いる色情報の位置とが、図２６（２）のサブピクセル型ベイア配列のi列目のサブピクセルとi+1列目のサブピクセルとの間を通る直線について、左右対称であるすると、座標[i+1,j]に位置するサブピクセルの斜め45度方向の類似度C45[i+1,j]および斜め135度方向の類似度C135[i+1,j]は、式171〜式184に記載のi-3〜i+4を、以下のように置き換えることによって算出できる。
【０２４７】
i-3→i+4,i-2→i+3,i-1→i+2,i→i+1,i+1→i,i+2→i-1,i+3→i-2,i+4→i-3。
補間処理部１７は、以上説明したようにして斜め方向の類似度を算出すると、その類似度に基づき、補間対象サブピクセルにおける斜め方向に対する類似性（以下、「斜め類似性」と称する。）を判定する（図２６Ｓ７）。そして、補間処理部１７は、第２の実施形態で斜め主要類似性の判定結果として指標DN[i_p,j]に値を設定した処理と同様に、斜め類似性の判定結果として指標DN[i,j]に値を設定する（図２６Ｓ８〜Ｓ１０）。
【０２４８】
なお、座標[i+1,j]に位置するサブピクセルの斜め類似性を示す指標DN[i+1,j]の設定は、上述したようにして算出される斜め45度方向の類似度C45[i+1,j]および斜め135度方向の類似度C135[i+1,j]を用いることによって、指標DN[i,j]と同様に設定することができる。
次に、補間処理部１７は、指標HV[i,j]および指標DN[i,j]が如何なる値であるかを判定し（図２６Ｓ１１）、第２の実施形態で補間対象画素の類似性の強さをcase1〜case9に分類した処理と同様に、補間対象サブピクセルの類似性の強さをcase1〜case9の何れかに分類する。
【０２４９】
そして、補間処理部１７は、このような分類の結果に応じて、以下のようにして座標[i,j]に位置する補間対象サブピクセルの緑の補間値G[i,j]を算出する。
case1のとき:G[i,j]=Gv45L:図２６Ｓ１２
case2のとき:G[i,j]=GvL:図２６Ｓ１３
case3のとき:G[i,j]=Gv135L:図２６Ｓ１４
case4のとき:G[i,j]=(Gv45L+Gh45L)/2:図２６Ｓ１５
case5のとき:G[i,j]=(GvL+GhL)/2:図２６Ｓ１６
case6のとき:G[i,j]=(Gv135L+Gh135L)/2:図２６Ｓ１７
case7のとき:G[i,j]=Gh45L:図２６Ｓ１８
case8のとき:G[i,j]=GhL:図２６Ｓ１９
case9のとき:G[i,j]=Gh135L:図２６Ｓ２０
ただし、GvL,Gv45L,Gv135L,GhL,Gh45L,Gh135Lは、第２の実施形態の式77〜式82のようにして算出される値である。
【０２５０】
なお、座標[i+1,j]に位置する補間対象サブピクセルの緑の補間値G[i+1,j]は、上述したように設定される指標HV[i+1,j]および指標DN[i,j]が如何なる値であるかを判定し、補間対象サブピクセルの類似性の強さをcase1〜case9の何れかに分類することによって、以下のように算出することができる。
case1のとき:G[i+1,j]=Gv45R
case2のとき:G[i+1,j]=GvR
case3のとき:G[i+1,j]=Gv135R
case4のとき:G[i+1,j]=(Gv45R+Gh45R)/2
case5のとき:G[i+1,j]=(GvR+GhR)/2
case6のとき:G[i+1,j]=(Gv135R+Gh135R)/2
case7のとき:G[i+1,j]=Gh45R
case8のとき:G[i+1,j]=GhR
case9のとき:G[i+1,j]=Gh135R
ただし、GvR,Gv45R,Gv135R,GhR,Gh45R,Gh135Rは、第２の実施形態の式83〜式88のようにして算出される値である。
【０２５１】
以上説明したように、第６の実施形態では、補間対象サブピクセル毎に、類似度を算出し、その類似度に基づく類似性の判定結果に応じて補間値を算出することができる。
なお、斜め45度方向の類似度C45[i,j]、C45[i+1,j]および斜め135度方向の類似度C135[i,j]、C135[i+1,j]は、式185および式186によって算出しても良い。
【０２５２】
C45[i,j]=C45[i+1,j]=
((|R[i+4,j-2]-R[i,j]|-|R[i-4,j+2]-R[i,j]|)/2
+(|R[i+5,j-2]-R[i+1,j]|-|R[i-3,j+2]-R[i+1,j]|)/2)/2 ・・・式185
C135[i,j]=C135[i+1,j]=
((|R[i-4,j-2]-R[i,j]|-|R[i+4,j+2]-R[i,j]|)/2
+(|R[i-3,j-2]-R[i+1,j]|-|R[i+5,j+2]-R[i+1,j]|)/2)/2 ・・・式186
ところで、上述した第１の実施形態ないし第６の実施形態の補間処理は、３色で構成されるサブピクセル型ベイア配列に限らず、サブピクセル構造を持つ３色以上のフィルタ配列に適用することも可能である。
【０２５３】
《第７の実施形態》
以下、第７の実施形態の動作を説明する。
図２７は、第７の実施形態の機能ブロック図である。
なお、第７の実施形態は、請求項５９ないし請求項６７に記載の補間処理プログラムを記録した記録媒体を用いて、パーソナルコンピュータによって補間処理を実行することに相当する。
【０２５４】
図２７において、機能が図１に示す機能ブロック図と同じものについては、同じ符号を付与して示し、構成の説明については省略する。
なお、図２７に示す電子カメラ２０と図１に示した電子カメラ１０との構成の相違点は、図２７の制御部２１と画像処理部２２とが図１の制御部１１と画像処理部１５とに代えて設けられ、図２７のインタフェース部２３が新たに設けられた点である。
【０２５５】
また、図２７において、パーソナルコンピュータ３０は、ＣＰＵ３１、インタフェース部３２、ハードディスク３３およびメモリ３４を有し、ＣＰＵ３１は、バスを介してインタフェース部３２、ハードディスク３３およびメモリ３４に接続される。
なお、パーソナルコンピュータ３０には、ＣＤ−ＲＯＭなどの記録媒体に記録された補間処理プログラム（前述した各実施形態の補間処理部１７と同様にして補間処理を実行する補間処理プログラム）が予めインストールされているものとする。すなわち、ハードディスク３３には、このような補間処理プログラムが実行可能な状態で格納されている。
【０２５６】
以下、図２７を参照して第７の実施形態の動作を説明する。
まず、電子カメラ２０では、図１に示した電子カメラ１０と同様にして生成された画像データが画像処理部２２に供給される。画像処理部２２は、画像データに補間処理以外の画像処理（例えば、階調変換処理など）を施し、記録部１６では、画像処理が施された画像データが画像ファイルの形式で記録される。
【０２５７】
このような画像ファイルは、インタフェース部２３を介してパーソナルコンピュータ３０に供給される。
パーソナルコンピュータ３０内のＣＰＵ３１は、インタフェース部３２を介して画像ファイルを取得すると、前述した補間処理プログラムを実行する。補間処理により各色成分の解像度が高められた画像データは、必要に応じて画像圧縮等を行ってから、ハードディスク３３などに記録され、最終的に、ディスプレイ、プリンタなどの各接続機器に応じた表色系データとして出力される。
【０２５８】
すなわち、第７の実施形態では、前述した各実施形態と同様の補間処理をパーソナルコンピュータ３０によって行うことができる。
【０２５９】
【発明の効果】
上述したように第１、第２７の発明では、画素ピッチでの大まかな類似性の強弱の判定を正確に行うことが可能であり、第２、第３および第５の発明では、画素ピッチでの大まかな類似性の強弱の判定とサブピクセルピッチでの細かな類似性の強弱の判定とが可能である。また、第４ないし第６、第２８の発明では、補間対象サブピクセルについて対象な距離に位置する色情報だけでなく、非対称な距離に位置する色情報を用いて類似性の強弱を判定する。さらに、類似度に基づき類似性の強弱を判定すれば、類似性の強弱を的確に判定することができ、補間対象サブピクセルの補間値を精度良く算出することができる。
【０２６０】
第７の発明では、主要類似性判定手段による判定の結果として類似性の強い方向が不明であると判断された場合にのみ、微細部類似性判定手段によって類似性の強弱を判定しても、類似性の強弱の判定の精度が低下することがなく、その上、主要類似性判定手段と微細部類似性判定手段とによる類似性の強弱の判定を全てのサブピクセルに対して行う場合と比べて、類似性の強弱を速やかに判定することができ、補間対象サブピクセルの補間値を速やかに算出することができる。
【０２６１】
第８ないし第１０の発明では、複数の色成分の色情報を用いて、類似性の強弱を判定するため、色調が変化する画像（異なる色成分間で色情報の比率が変化する画像）であっても、類似性を的確に判定することができる。また、これらの発明では、複数の色成分の色情報として、補間対象サブピクセルの色情報やその近傍に存在する色情報を用い、類似性の強弱を判定することが可能であるため、空間周波数が高い画像であっても、類似性を的確に判定することができる。したがって、これらの発明によれば、補間処理を行うべき画像に「色調が変化する部分」や「空間周波数が高い部分」が混在する場合であっても、画像全体の調和を保ちつつ、精度良く補間処理を行うことができる。
【０２６２】
第１１、第２３の発明は、補間対象サブピクセルが属する局所的な領域における画像の特徴に応じて、異色間類似度と同色間類似度とを使い分けたり、異色間類似度と同色間類似度とを加重加算する際の加重比率を変えるため、類似性の強弱の判定に適した類似度を算出することができると共に、複数の色情報を用いて異色間類似度および同色間類似度の算出することによって、類似度の算出精度を上げることができる。したがって、補間量を高い精度で算出することができる。
【０２６３】
第１２、第２４の発明では、異色間類似度を同色間類似度よりも短い距離間隔で存在する色情報を用いて算出するので、異色間類似度に空間周波数が高い画像の類似性が反映される。すなわち、異色間類似度を用いて類似性の強弱を判定する際や、色彩に関する情報として異色間類似度を用いて補間対象サブピクセルが属する局所的な領域における画像の特徴を抽出する際に、高周波部分の解像能力が効果的に引き出すことができる。そのため、類似性の強弱の判定精度を上げることができ、補間量の精度が高められる。
【０２６４】
第１３の発明は、異色間類似度および同色間類似度の算出に多くの色成分を用いた類似度成分を生成することにより、類似度を高い精度で算出することができ、補間量の精度が高められる。
第１４の発明は、色彩に関する情報を用いることによって、補間対象サブピクセルが属する局所的な領域における画像の特徴を抽出するので、異色間類似度と同色間類似度との使い分けや、異色間類似度と同色間類似度との加重比率の設定に際し、適切な指標に基づいて画像の特徴抽出が行える。そのため、類似性の強弱の判定に適した類似度を算出することができ、補間量の精度が高められる。
【０２６５】
第１５、第１６の発明は、補間対象サブピクセルが属する局所的な領域における画像の特徴を抽出する際、色彩に関する情報として異色間類似度を用いるので、類似性の強弱の判定のために異色間類似度を算出してしまえば、色彩に関する情報を改めて求める必要がない。すなわち、補間処理全体を効率良く行うことができる。
【０２６６】
第１７、第２５の発明では、補間対象サブピクセルと周辺のサブピクセルとの間で、類似性が強い方向の連続性を考慮しつつ、補間対象サブピクセルの類似度を算出することができ、第１８、第２６の発明では、ノイズなどによって発生する類似性の誤判定を低減することができる。すなわち、これらの発明では、類似性の強弱の判定精度を上げることができるので、補間量の精度が高められる。
【０２６７】
第１９の発明では、補間対象サブピクセルにおける少なくとも２つの方向に対する類似度に、特定の１種類の色成分の色情報や複数種類の色成分の色情報を反映させることができ、第２０の発明では、補間対象サブピクセルにおける少なくとも２つの方向に対する類似度に、複数種類の色成分の色情報を反映させることができる。また、第２１の発明では、補間対象サブピクセルにおける少なくとも２つの方向に対する類似度に、複数種類の色成分の色情報を的確に反映させることができ、第２２の発明では、各々の類似度成分に対する最短距離の相違に影響されることなく、適切な類似度を算出することができる。したがって、これらの発明によれば、補間対象サブピクセルを含む局所領域の画像の特徴を反映しつつ、類似性の強弱を判定することができるため、画像全体の調和を崩すことなく補間値を算出することができる。
【図面の簡単な説明】
【図１】第１の実施形態ないし第６の実施形態に対応する電子カメラの機能ブロック図である。
【図２】第１の実施形態における補間処理部の動作フローチャート（１）である。
【図３】第１の実施形態における補間処理部の動作フローチャート（２）である。
【図４】周辺加算方法を説明する図である。
【図５】微細類似度成分を算出する際に用いる色情報の位置を示す図である。
【図６】色差補間で用いる色情報の位置を示す図である。
【図７】色差の線形補間を説明する図である。
【図８】色差を中央値で補間する処理を説明する図である。
【図９】 median処理の範囲を説明する図である。
【図１０】倍率色収差の影響を説明する図である。
【図１１】倍率色収差による過補正を説明する図である。
【図１２】第２の実施形態における補間処理部の動作フローチャートである。
【図１３】第２の実施形態における補間処理部の動作フローチャート（続き）である。
【図１４】 (HV[i_p,j],DN[i_p,j])の値に対応する類似性の強い方向を示す図である。
【図１５】緑の補間値を算出する際に用いる色情報の位置を示す図（１）である。
【図１６】緑の補間値を算出する際に用いる色情報の位置を示す図（２）である。
【図１７】緑の補間値を算出する際に用いる色情報の位置を示す図（３）である。
【図１８】倍率色収差の影響が解消される様子を説明する図である。
【図１９】第３の実施形態における補間処理部の動作フローチャート（１）である。
【図２０】第３の実施形態における補間処理部の動作フローチャート（２）である。
【図２１】第３の実施形態における補間処理部の動作フローチャート（３）である。
【図２２】類似度と類似性の強弱の関係を示す図である。
【図２３】輝度を生成するためのフィルタを示す図である。
【図２４】サブピクセル型ベイア配列における周辺加算方法を説明する図である。
【図２５】第５の実施形態における補間処理部の動作フローチャートである。
【図２６】第６の実施形態における補間処理部の動作フローチャートである。
【図２７】第７の実施形態における機能ブロック図である。
【図２８】撮像素子における色成分の配列を示す図である。
【符号の説明】
１０、２０ 電子カメラ
１１、２１ 制御部
１２ 撮影光学系
１３ 撮像部
１４ Ａ／Ｄ変換部
１５、２２ 画像処理部
１６ 記録部
１７ 補間処理部
２３、３２ インタフェース部
３０ パーソナルコンピュータ
３１ ＣＰＵ
３３ ハードディスク
３４ メモリ[0001]
BACKGROUND OF THE INVENTION
The present invention performs an interpolation process for determining an interpolation value corresponding to a color component and a luminance component that are missing in each sub-pixel, with respect to image data generated by an imaging device in which each pixel is divided into two sub-pixels. The present invention relates to an interpolation processing device to be performed and a computer-readable recording medium on which an interpolation processing program for realizing the interpolation processing by a computer is recorded.
[0002]
[Prior art]
Some electronic cameras generate color image data by an image sensor in which color filters corresponding to color components of three colors (RGB: red, green, and blue) are arranged in a Bayer array. In such an electronic camera, since only color information of one color component is output from each pixel of the image sensor, it is necessary to perform an interpolation process in order to obtain color information of all the color components in units of pixels.
[0003]
Therefore, the present applicant has applied for an invention that can perform an interpolation process with high accuracy even for an image including a high-frequency component as Japanese Patent Application Nos. 11-145273 and 11-145473. According to these inventions, a high-definition image can be obtained because an interpolation process can be performed with high accuracy even for an image whose color changes at a pixel pitch which is the limit of the resolution of the image sensor.
[0004]
[Problems to be solved by the invention]
By the way, the resolution of the image sensor in which color components are arranged in a Bayer array as shown in FIG. 28 (1) is obtained by dividing each pixel into two areas as shown in FIG. Improve by getting In the following, each region obtained by dividing one pixel into two is referred to as a “subpixel”, and an arrangement of color components as shown in FIG. 28 (2) is referred to as a “subpixel type Bayer arrangement”.
[0005]
As shown in FIG. 28 (2), an electronic camera including image sensors arranged in a sub-pixel type Bayer array can obtain a higher-definition image by performing interpolation processing in units of sub-pixels.
However, in the sub-pixel type Bayer array, when color components are captured in units of sub-pixels, the arrangement of the color components in the horizontal direction starting from an arbitrary sub-pixel is asymmetric. Therefore, the above-described invention cannot be easily applied to the image data generated by the image pickup elements arranged in the sub-pixel type Bayer array.
[0006]
  Therefore,1st to 26thAn object of the present invention is to provide an interpolation processing apparatus that can accurately perform interpolation processing in units of sub-pixels on image data generated by an imaging device in which each pixel is divided into two sub-pixels.27th or 28thIt is an object of the present invention to provide a recording medium on which an interpolation processing program capable of performing interpolation processing with accuracy on a sub-pixel basis for such image data is recorded.
[0007]
[Means for Solving the Problems]
  In order to solve the above-described problem, the first inventionIn the interpolation processing device, one color component is associated with each pixel of a plurality of pixels arranged two-dimensionally, and image data generated by an imaging device in which each pixel is divided into two sub-pixels, In an interpolation processing apparatus that performs processing for determining an interpolation value corresponding to color information of a color component that is missing in each sub-pixel, a value obtained by collectively gathering color information output from two sub-pixels constituting each pixel Main similarity determination means that determines pixel output color information corresponding to pixel color information, and determines the strength of similarity of interpolation target sub-pixels in at least two directions using the pixel output color information, and the main similarity According to the strength of the similarity determined by the determination means, color information of the sub-pixel used when calculating the interpolation value of the interpolation target sub-pixel is selected, and the color information Using characterized by comprising an interpolation value calculating means for calculating an interpolation value of the interpolation target subpixel.
That is, the first invention can determine the strength of similarity at the pixel pitch. In the first invention, it is possible to apply an oblique direction in addition to the vertical direction and the horizontal direction as at least two directions in which the strength of similarity is determined.
[0008]
The second invention is the first invention., The subpixel output color information is used as the color information output from the subpixel, and the similarity of the subpixels to be interpolated in at least two directions is determined using the subpixel output color information. And the interpolation value calculation means calculates the interpolation value of the sub-pixel to be interpolated according to the strength of the similarity determined by the main similarity determination means and the fine part similarity determination means. The color information of the subpixel to be used is selected, and the interpolation value of the subpixel to be interpolated is calculated using the color information.
[0009]
  By the way, in the image data generated by the image pickup element in which each pixel is divided into two sub-pixels, it is possible to roughly determine the direction of strong similarity by determining the strength of similarity at the pixel pitch. However, in order to determine in detail the direction of strong similarity, it is desirable to determine the strength of similarity at the sub-pixel pitch.
  SecondIn the invention, the fine part similarity determination means determines the strength of the similarity of the sub-pixels to be interpolated using the sub-pixel output color information, so that the strength of the similarity at the sub-pixel pitch can be determined.
[0010]
  Third inventionIsSecond inventionThe main similarity determination means determines the strength of the similarity using pixel output color information of a pixel located at a symmetrical distance with respect to the pixel to which the interpolation target subpixel belongs, and the fine part similarity determination means The similarity is determined using subpixel subpixel output color information located at an asymmetric distance with respect to the interpolation target subpixel.
[0011]
  That is,ThirdIn the invention, the main similarity determination means determines the strength of the similarity using the pixel output color information of the pixel located at a symmetrical distance with respect to the pixel to which the interpolation target sub-pixel belongs, thereby determining the similarity at the pixel pitch. Can determine strength and weakness.
  By the way, in the image data generated by the imaging device in which each pixel is divided into two subpixels, for example, when each pixel is divided along a straight line in the vertical direction, an arbitrary subpixel is set as the starting point. The arrangement of the horizontal color components is asymmetric.In the third invention, even if the arrangement of the color components is asymmetric with respect to an arbitrary direction, the fine portion similarity determination means outputs the subpixel output color information of the subpixel located at an asymmetric distance with respect to the interpolation target subpixel. By using this method to determine the strength of similarity, it is possible to determine the strength of similarity at the sub-pixel pitch.
[0012]
4th inventionIn the interpolation processing apparatus, one color component is associated with each pixel of a plurality of pixels arranged two-dimensionally, and image data generated by an imaging device in which each pixel is divided into two sub-pixels. In an interpolation processing apparatus that performs processing for determining an interpolation value corresponding to color information of a color component that is missing in each sub-pixel, interpolation is performed in at least two directions using color information that is located at a symmetrical distance with respect to the interpolation target sub-pixel. Using main similarity determination means for determining the strength of similarity of the target subpixel and color information located at an asymmetric distance with respect to the interpolation target subpixel, the strength of similarity of the interpolation target subpixel in at least two directions is determined. Of the similarity determined by the fine part similarity determining means, the main similarity determining means and the fine part similarity determining means. Interpolation value calculating means for selecting the color information of the subpixel used when calculating the interpolation value of the interpolation target subpixel according to the weakness, and calculating the interpolation value of the interpolation target subpixel using the color information. It is characterized by that.
[0013]
  5th inventionIs4th inventionIn the above, the main similarity determination unit outputs color information output from two sub-pixels constituting a pixel located at a target distance for a pixel to which the interpolation target sub-pixel belongs as color information located at the target distance. Using the grouped values, the fine part similarity determination means uses, as the color information located at the asymmetric distance, color information output from the subpixel located at the asymmetric distance with respect to the interpolation target subpixel. Features.
[0014]
  6th inventionIs4th inventionThe main similarity determination means uses the color information output from the sub-pixel located at a symmetric distance with respect to the interpolation target sub-pixel as the color information located at the target distance, and the fine-part similarity determination means Is characterized in that color information output from a subpixel located at an asymmetric distance with respect to the interpolation target subpixel is used as the color information located at the asymmetric distance.
[0015]
  7th inventionIsAny one of the second to sixth inventionsThe fine part similarity determining means determines the strength of the similarity when it is determined that the direction of strong similarity is unknown as a result of the determination by the main similarity determining means. .
  By the way, in the image data generated by the imaging device in which each pixel is divided into two subpixels, even when the direction of strong similarity cannot be determined by determining the strength of similarity at the pixel pitch, It is possible to determine the direction of strong similarity by determining the strength of the similarity at the sub-pixel pitch.
[0016]
  7thIn the invention, the determination of the strength of the similarity at the sub-pixel pitch by the fine portion similarity determination means can supplement the determination of the similarity strength at the pixel pitch by the main similarity determination means, so the determination by the main similarity determination means Only when it is determined that the direction of strong similarity is unknown as a result of the above, even if the similarity of the similarity is determined by the fine portion similarity determination means, the accuracy of the determination of similarity may be reduced. Absent.
[0017]
  Eighth inventionIsAny one of the first to seventh inventionsIn the case where the image data is represented by a color system comprising a plurality of color components, the main similarity determination means calculates the similarity in at least two directions in the interpolation target sub-pixel, and the color information of the plurality of color components. And the strength of the similarity of the interpolation target sub-pixels in at least two directions is determined based on the similarity.
[0018]
  Ninth inventionIsEighth inventionThe image data is represented by a color system composed of first to third color components, the first color component has a higher spatial frequency than the second color component and the third color component, and the first color component When determining an interpolated value corresponding to the color information of the first color component for the sub-pixel lacking
(1) a similarity component using color information of the first color component and color information of the second color component;
(2) a similarity component using color information of the first color component and color information of the third color component;
(3) a similarity component using only color information of the first color component;
(4) a similarity component using only the color information of the second color component;
(5) a similarity component using only the color information of the third color component;
Among the plurality of similarity components, the similarity for at least two directions in the interpolation target subpixel is calculated using a combination including the color information of the first to third color components.
[0019]
  10th inventionIsNinth inventionThe main similarity determination means calculates the similarity in at least two directions in the interpolation target sub-pixel by weight-adding the similarity components belonging to the combination among the plurality of similarity components for each direction. It is characterized by.
[0020]
  11th departureAkira1st to 7th inventionWhen the image data is represented by a color system composed of a plurality of color components, the main similarity determination means uses a plurality of colors selected from the color information of the sub-pixel to be interpolated and the color information existing in the vicinity thereof. Based on information
(1) The similarity between different colors calculated using color information of different color components;
(2) Similarity similarity between colors calculated using color information of the same color component
The weighting ratio when calculating the two types of similarity and switching to one of the similarities according to the characteristics of the image in the local region to which the interpolation target subpixel belongs, or weighting each similarity , The similarity in at least two directions in the interpolation target subpixel is calculated, and the similarity strength of the interpolation target subpixel in at least one direction is determined based on the similarity.
[0021]
  12th inventionIsEleventh inventionThe main similarity determination unit calculates the similarity between different colors using color information existing at a distance interval shorter than the similarity between the same colors..
[0022]
  In a thirteenth aspect based on the eleventh or twelfth aspect, the image data is represented by a color system composed of first to third color components, and the first color component is the second color component and the third color. When determining an interpolated value corresponding to the color information of the first color component for a sub-pixel having a higher spatial frequency than the component and lacking the first color component, the main similarity determination means, as the similarity between different colors,
(1) A different color similarity component composed of color information of the first color component and color information of the second color component;
(2) A different color similarity component comprising color information of the first color component and color information of the third color component;
Calculating the similarity including at least one of
As the similarity between the same colors,
(1) The same-color similarity component consisting only of color information of the first color component;
(2) The same-color similarity component consisting only of color information of the second color component;
(3) The same-color similarity component consisting only of color information of the third color component;
Calculate the similarity including at least one of
It is characterized by that.
[0023]
  In a fourteenth aspect based on the eleventh aspect, the main similarity determination unit uses information about a color as a feature of an image in a local region to which the interpolation target subpixel belongs, and the information about the color is used as a reference. It is determined whether the image in the local region is an achromatic portion or a chromatic portion. If the image in the local region is an achromatic portion, the similarity in at least two directions in the sub-pixel to be interpolated is determined. The degree of similarity between different colors is selected as the degree of similarity, or the weighted ratio of the degree of similarity between different colors when weighted addition of the degree of similarity between different colors and the degree of similarity between same colors is set to be higher than the weighted ratio of the degree of similarity between same colors When the image in the local region is a chromatic part, the similarity between the same colors is selected as the similarity in at least two directions in the sub-pixel to be interpolated. Or, wherein the larger than the weighted ratio of the different color similarity between the weighted ratio of the of identity color similarity between the time of weighted addition of said same color Similarity between the different color similarity between.
[0024]
  According to a fifteenth aspect, in the fourteenth aspect, the main similarity determination unit uses the similarity between different colors as information on the color.
[0025]
  In a sixteenth aspect based on the fourteenth or fifteenth aspect, the main similarity determination means determines that the image in the local area is similar when the similarity between different colors shows a strong similarity in at least one direction. Is determined to be an achromatic portion, and in other cases, it is determined that an image in the local region is a chromatic portion.
[0026]
  In a seventeenth aspect based on any one of the eighth to sixteenth aspects, the main similarity determination means determines not only the interpolation target subpixel but also the interpolation target subpixel as the similarity in at least two directions of the interpolation target subpixel. The similarity in at least two directions calculated for the peripheral sub-pixels is used.
[0027]
  By the way, in many cases, there is some relationship (for example, continuity of change in color information) between the interpolation target sub-pixel and the surrounding sub-pixels.
In the seventeenth invention, not only the similarity in the interpolation target subpixel but also the similarity in the surrounding subpixels are used together to determine the strength of the similarity at the pixel pitch. The association with the sub-pixel can be reflected.
[0028]
  In an eighteenth aspect based on the eighth or eleventh aspect, the main similarity determination means has similar degrees of similarity in each direction when the difference in similarity between the directions is smaller than a predetermined threshold. It is characterized by determining.
  Therefore, according to the eighteenth aspect, the influence of noise when determining similarity can be reduced.
[0029]
  In a nineteenth aspect based on any one of the second to seventh aspects, the fine portion similarity determination means determines the similarity in at least two directions in the interpolation target sub-pixel, as color information of one or more color components. And the strength of similarity of the sub-pixels to be interpolated in at least two directions is determined based on the similarity.
[0030]
  That is, in the nineteenth aspect, the strength of the similarity at the subpixel pitch is determined based on the similarity calculated using the color information of one specific type of color component or the color information of a plurality of types of color components.
[0031]
  In a twentieth aspect according to the nineteenth aspect, any one of the first to nth (n ≧ 2) color components is associated with one pixel, and the spatial frequency is lower than that of the first color component. When the interpolated value of the first color component is supplemented to the sub-pixel constituting the pixel associated with the second color component, the fine part similarity determination means includes
(1) Among the subpixels that output the color information of the second color component, a similarity component indicating a difference in the color information of the second color component between the subpixels located at the shortest distance;
(2) Of the subpixels that output the color information of the first color component, a similarity component indicating a difference in color information of the first color component between the subpixels located at the shortest distance;
(3) Of the subpixel that outputs the color information of the first color component and the subpixel that outputs the color information of the second color component, the first color component and the second color component of the subpixel located at the shortest distance; Similarity component indicating the difference in color information between
The similarity in at least two directions in the interpolation target sub-pixel is calculated using at least one similarity component.
[0032]
  That is, in the twentieth aspect, by using a plurality of types of similarity components, the color information of the plurality of types of color components can be reflected in the similarity in at least two directions in the interpolation target subpixel.
[0033]
  In a twenty-first aspect based on the twentieth aspect, the fine portion similarity determination unit performs weighted addition of the respective similarity components in each direction, and calculates the similarity in at least two directions in the interpolation target subpixel. It is characterized by doing.
That is, in the twenty-first aspect, the color information of a plurality of types of color components can be accurately reflected in the similarity in at least two directions in the interpolation target subpixel.
[0034]
  In a twenty-second aspect based on the twentieth or twenty-first aspect, the fine portion similarity determination unit calculates each similarity component between subpixels when calculating the similarity with respect to each direction in the interpolation target subpixel. It is characterized by being standardized according to the distance.
[0035]
  By the way, in the image data generated by the imaging device in which each pixel is divided into two sub-pixels, “the shortest distance of the sub-pixels that output the color information of the second color component” and “the color of the first color component” The “shortest distance between subpixels that output information” and “the shortest distance between subpixels that output color information of the first color component and subpixels that output color information of the second color component” are often different. Such a difference in the shortest distance affects the value of each similarity component.
[0036]
  In the twenty-second aspect, when calculating the similarity in each direction, each similarity component is normalized and used according to the distance between the sub-pixels, so that it is appropriate without being affected by the difference in the shortest distance. A similar degree of similarity can be calculated.
[0037]
  In a twenty-third aspect of the present invention, in any one of the second to seventh aspects, when the image data is represented by a color system comprising a plurality of color components, the fine portion similarity determination means Based on a plurality of color information selected from color information and color information existing in the vicinity thereof,
(1) The similarity between different colors calculated using color information of different color components;
(2) Similarity similarity between colors calculated using color information of the same color component
The weighting ratio when calculating the two types of similarity and switching to one of the similarities according to the characteristics of the image in the local region to which the interpolation target subpixel belongs, or weighting each similarity , The similarity in at least two directions in the interpolation target subpixel is calculated, and the similarity strength of the interpolation target subpixel in at least one direction is determined based on the similarity.
[0038]
  In a twenty-fourth aspect based on the twenty-third aspect, the fine part similarity determination unit calculates the similarity between different colors using color information existing at a distance interval shorter than the similarity between the same colors. And
[0039]
  In a twenty-fifth aspect based on any one of the nineteenth to twenty-fourth aspects, the fine part similarity determining means determines not only the interpolation target subpixel but also the interpolation target as the similarity to at least two directions in the interpolation target subpixel. It is characterized by using the similarity in at least two directions calculated for subpixels around the subpixel.
  In the twenty-fifth aspect of the present invention, not only the similarity in the interpolation target subpixel but also the similarity in the surrounding subpixels are used together to determine the strength of similarity at the subpixel pitch. It is possible to reflect the association with the sub-pixels of.
[0040]
  In a twenty-sixth aspect based on the nineteenth or twenty-third aspect, the fine portion similarity determination means has the same degree of similarity in each direction when the difference in similarity between the directions is smaller than a predetermined threshold. It is determined that it exists.
  Therefore, according to the invention of claim 26, it is possible to reduce the influence of noise when determining similarity.
[0041]
  According to a twenty-seventh aspect of the present invention, an image generated by an imaging device in which one color component is associated with each pixel of a plurality of pixels arranged two-dimensionally and each pixel is divided into two subpixels. In a recording medium on which an interpolation processing program for realizing an interpolation processing for determining interpolation values corresponding to the color information of color components missing in each sub-pixel is recorded with respect to the data, the two constituting each pixel A value obtained by grouping the color information output from the sub-pixel is set as pixel output color information corresponding to the color information of the pixel, and the similarity strength of the sub-pixel to be interpolated in at least two directions is expressed as the pixel output color information. The main similarity determination procedure to be used and the complement of the sub-pixel to be interpolated according to the strength of the similarity determined by the main similarity determination procedure. Select the color information of the subpixels to be used in calculating the value, characterized in that to perform the interpolation value calculation procedure of calculating the interpolation value of the interpolation target subpixel computer using the color information.
[0042]
  The recording medium of the twenty-eighth aspect of the invention is generated by an imaging device in which one color component is associated with each pixel of a plurality of pixels arranged two-dimensionally, and each pixel is divided into two subpixels. In a recording medium on which an interpolation processing program for realizing an interpolation process for determining the interpolation value corresponding to the color information of the color component missing in each subpixel is recorded with respect to the image data to be recorded, the interpolation target subpixel is symmetric. A main similarity determination procedure for determining the strength of similarity of the interpolation target subpixel in at least two directions using color information located at a certain distance, and color information located at an asymmetric distance with respect to the interpolation target subpixel, The fine part similarity determination procedure for determining the strength of the similarity of the interpolation target sub-pixels in at least two directions; In accordance with the strength of the similarity determined by the sex determination procedure and the fine part similarity determination procedure, the color information of the subpixel used when calculating the interpolation value of the interpolation target subpixel is selected, and the color information is selected. And an interpolation value calculation procedure for calculating an interpolation value of an interpolation target sub-pixel using the computer.
[0088]
DETAILED DESCRIPTION OF THE INVENTION
  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
  FIG. 1 is a functional block diagram of an electronic camera corresponding to the first to sixth embodiments..
[0089]
In FIG. 1, the electronic camera 10 includes a control unit 11, a photographing optical system 12, an imaging unit 13, an A / D conversion unit 14, an image processing unit 15, and a recording unit 16.
Further, the image processing unit 15 includes an interpolation processing unit (for example, a one-chip microprocessor dedicated to interpolation processing) 17. In FIG. 1, only the interpolation processing unit 17 is shown in the image processing unit 15 for the sake of simplicity, but other images such as γ gradation conversion processing are included in the image processing unit 15. A functional block that performs processing may be provided.
[0090]
Further, the imaging unit 13 includes an imaging device in which color components are arranged in a sub-pixel type Bayer array as shown in FIG. In FIG. 28, the coordinates of pixels and subpixels are shown using i and j, but in order to distinguish the coordinates on the Bayer array from the coordinates on the subpixel type Bayer array, FIG. The subscript p is given to the coordinate i in the Bayer array.
[0091]
In FIG. 1, the control unit 11 is connected to an imaging unit 13, an A / D conversion unit 14, an image processing unit 15, and a recording unit 16. In addition, the optical image acquired by the photographic optical system 12 is formed on the image sensor in the imaging unit 13. The output of the imaging unit 13 is quantized by the A / D conversion unit 14 and supplied to the image processing unit 15 as image data. The image data supplied to the image processing unit 15 is subjected to interpolation processing by the interpolation processing unit 17, subjected to image compression as necessary, and then recorded via the recording unit 16. The image data in which the resolution of each color component is increased by the interpolation processing is finally output as color system image data corresponding to each connected device such as a display and a printer.
[0092]
In the first to sixth embodiments, the interpolation processing unit 17 performs an interpolation process (hereinafter referred to as “G interpolation process”) that supplements a green interpolation value to a sub-pixel that lacks a green component. After this, an interpolation process (hereinafter referred to as “RB interpolation process”) is performed to supplement the red interpolation value and the blue interpolation value for the sub-pixels lacking the red component and the blue component. Therefore, two subpixels located at the coordinates [i, j], [i + 1, j] in FIG. 28 (2) are set as interpolation target subpixels in the G interpolation process.
[0093]
In the first to third embodiments, the color information of the pixels on the Bayer array in FIG. 28 (1) is displayed on the sub-pixel type Bayer array in FIG. 28 (2) corresponding to each pixel. The average value of the color information of the two subpixels located is used. For example, the coordinates [i in FIG._p, j], the color information R [i_p, j] is the color information R [i, j], R [i + 1, j] of two subpixels located at the coordinates [i, j], [i + 1, j] in FIG. It corresponds to the average value of.
[0094]
Furthermore, in the first to third embodiments, the pixels on the Bayer array corresponding to the two interpolation target sub-pixels are referred to as “interpolation target pixels”.
In the G interpolation processing described later, since the green interpolation value can be calculated regardless of the color component type (red or blue) of the interpolation target sub-pixel, R and B in FIG. The color information of two sub-pixels to be interpolated is represented by Z [i, j], Z [i + 1, j], and the color information of other subpixels and pixels is similarly represented. .
[0095]
<< First Embodiment >>
2 and 3 are operation flowcharts of the interpolation processing unit 17 in the first embodiment. FIG. 2 shows the operation of the interpolation processing unit 17 in the G interpolation processing, and FIG. 3 shows the interpolation in the RB interpolation processing. The operation of the processing unit 17 is shown.
Hereinafter, the operation of the first embodiment will be described. Here, the operation of the interpolation processing unit 17 will be described with reference to FIGS. 2 and 3.
[0096]
First, in the G interpolation process, the interpolation processing unit 17 indicates “rough main similarity Cv [i in the vertical direction” indicating a rough vertical similarity in the interpolation target pixels on the Bayer array._p, j] ”and“ lateral main similarity Ch [i_p, j] ”(S1 in FIG. 2).
The main similarity is a value for determining a wide-area similarity with the surroundings, and it is important to use as much color information as possible and capture the similarity by continuously changing multi-value information. . The main similarity calculated as described below can determine the detailed similarity up to the pixel pitch of the Bayer array while maintaining the harmony of the entire image.
[0097]
Hereafter, the main similarity Cv [i in the vertical direction_p, j] and the horizontal similarity Ch [i_p, j] will be described in detail.
First, the interpolation processing unit 17 calculates a plurality of types of main similarity components for the vertical direction and the horizontal direction defined by the following Expressions 10 to 21.
Main similarity component between GG in the vertical direction:
Cv1 [i_p, j] = | G [i_p, j-1] -G [i_p, j + 1] |
Horizontal similarity component between GG:
Ch1 [i_p, j] = | G [i_p-1, j] -G [i_p+ 1, j] |
Main similarity component between BB (RR) in the vertical direction:
Cv2 [i_p, j] = (| Z [i_p-1, j-1] -Z [i_p-1, j + 1] |
+ | Z [i_p+ 1, j-1] -Z [i_p+ 1, j + 1] |) / 2 (12)
Main similarity component between horizontal BB (RR):
Ch2 [i_p, j] = (| Z [i_p-1, j-1] -Z [i_p+ 1, j-1] |
+ | Z [i_p-1, j + 1] -Z [i_p+ 1, j + 1] |) / 2 Equation 13
Main similarity component between vertical RR (BB):
Cv3 [i_p, j] = (| Z [i_p, j-2] -Z [i_p, j] | + | Z [i_p, j + 2] -Z [i_p, j] |) / 2 ・ ・ ・ Formula 14
Main similarity component between horizontal RR (BB):
Ch3 [i_p, j] = (| Z [i_p-2, j] -Z [i_p, j] | + | Z [i_p+ 2, j] -Z [i_p, j] |) / 2 ・ ・ ・ Formula 15
Major similarity component between longitudinal GR (GB):
Cv4 [i_p, j] = (| G [i_p, j-1] -Z [i_p, j] | + | G [i_p, j + 1] -Z [i_p, j] |) / 2 ・ ・ ・ Equation 16
Main similarity component between horizontal GR (GB):
Ch4 [i_p, j] = (| G [i_p-1, j] -Z [i_p, j] | + | G [i_p+ 1, j] -Z [i_p, j] |) / 2 ・ ・ ・ Equation 17
Main similarity component between vertical BG (RG):
Cv5 [i_p, j] = (| Z [i_p-1, j-1] -G [i_p-1, j] | + | Z [i_p-1, j + 1] -G [i_p-1, j] |
+ | Z [i_p+ 1, j-1] -G [i_p+ 1, j] | + | Z [i_p+ 1, j + 1] -G [i_p+ 1, j] |) / 4 ・ ・ ・ Equation 18
Main similarity component between lateral BG (RG):
Ch5 [i_p, j] = (| Z [i_p-1, j-1] -G [i_p, j-1] | + | Z [i_p-1, j + 1] -G [i_p, j + 1] |
+ | Z [i_p+ 1, j-1] -G [i_p, j-1] | + | Z [i_p+ 1, j + 1] -G [i_p, j + 1] |) / 4 Equation 19
Main similarity component between luminance in vertical direction:
Cv6 [i_p, j] = (| Y [i_p, j-1] -Y [i_p, j] | + | Y [i_p, j + 1] -Y [i_p, j] |) / 2 ・ ・ ・ Equation 20
Main similarity component between luminance in horizontal direction:
Ch6 [i_p, j] = (| Y [i_p-1, j] -Y [i_p, j] | + | Y [i_p+ 1, j] -Y [i_p, j] |) / 2 ・ ・ ・ Formula 21
However, in Equation 20 and Equation 21, Y [i_p, j] is a value calculated by Equation 22, and is a filtering that averages the color information of the color components of neighboring pixels located in the vicinity of the interpolation target pixel at a ratio of R: G: B = 1: 2: 1 This corresponds to the luminance generated by the processing.
[0098]
Y [i_p, j] = (4 ・ A [i_p, j] +2 ・ (A [i_p, j-1] + A [i_p, j + 1] + A [i_p-1, j] + A [i_p+ 1, j])
+ A [i_p-1, j-1] + A [i_p-1, j + 1] + A [i_p+ 1, j-1] + A [i_p+ 1, j + 1]) / 16 ・ ・ ・ Equation 22
However, in Equation 22, A [i_p, j] represents arbitrary color information on the Bayer array, and takes a value of G or Z depending on the arrangement location.
[0099]
In addition, although the element of the similarity which comprises each main similarity component mentioned above is calculated using the absolute value of a difference, you may calculate by the square of an absolute value, a power, etc. Next, as shown in the following Expression 23 and Expression 24, the interpolation processing unit 17 uses the weighting coefficients a1, a2, a3, a4, a5, and a6 to add a plurality of types of main similarity components for each direction. Weighted addition.
[0100]
Cv0 [i_p, j] = (a1 ・ Cv1 [i_p, j] + a2 ・ Cv2 [i_p, j] + a3 ・ Cv3 [i_p, j] + a4 ・ Cv4 [i_p, j]
+ a5 ・ Cv5 [i_p, j] + a6 ・ Cv6 [i_p, j]) / (a1 + a2 + a3 + a4 + a5 + a6) ... Equation 23
Ch0 [i_p, j] = (a1 ・ Ch1 [i_p, j] + a2 ・ Ch2 [i_p, j] + a3 ・ Ch3 [i_p, j] + a4 ・ Ch4 [i_p, j]
+ a5 ・ Ch5 [i_p, j] + a6 ・ Ch6 [i_p, j]) / (a1 + a2 + a3 + a4 + a5 + a6) ... Equation 24
In Expression 23 and Expression 24, the ratio of the weighting coefficients a1, a2, a3, a4, a5, a6 is, for example, “a1: a2: a3: a4: a5: a6 = 2: 1: 1: 4: 4:12 "and so on.
[0101]
Cv0 [i calculated in this way_p, j] and Ch0 [i_p, j] is the main similarity Cv [i in the vertical direction of the pixel to be interpolated as it is._p, j] and the horizontal similarity Ch [i_p, j], the calculation and weighted addition of a plurality of types of main similarity components in the vertical direction and the horizontal direction are performed not only on the interpolation target pixel but also on the surrounding pixels. A value obtained by performing weighted addition is weighted for each direction (hereinafter referred to as “peripheral addition”), and the main similarity Cv [i in the vertical direction of the interpolation target pixel is calculated._p, j] and the horizontal similarity Ch [i_p, j] is shown.
[0102]
That is, the interpolation processing unit 17 performs the result of weighted addition for each direction of the main similarity component between the interpolation target pixel and the surrounding pixels (Cv0 [i_p, j], Cv0 [i_p-1, j-1], Cv0 [i_p-1, j + 1], Cv0 [i_p+ 1, j-1], Cv0 [i_p+ 1, j + 1], etc.) are weighted and added as in the following << peripheral addition method 1 >> or << peripheral addition method 2 >>, and the vertical main similarity Cv [i_p, j] and the horizontal similarity Ch [i_p, j].
[0103]
<< Peripheral addition method 1 >>
Cv [i_p, j] = (4 ・ Cv0 [i_p, j] + Cv0 [i_p-1, j-1] + Cv0 [i_p-1, j + 1]
+ Cv0 [i_p+ 1, j-1] + Cv0 [i_p+ 1, j + 1]) / 8 ・ ・ ・ Equation 25
Ch [i_p, j] = (4 ・ Ch0 [i_p, j] + Ch0 [i_p-1, j-1] + Ch0 [i_p-1, j + 1]
+ Ch0 [i_p+ 1, j-1] + Ch0 [i_p+ 1, j + 1]) / 8 ・ ・ ・ Equation 26
<< Peripheral addition method 2 >>
Cv [i_p, j] = (4 ・ Cv0 [i_p, j] +2 ・ (Cv0 [i_p-1, j-1] + Cv0 [i_p+ 1, j-1] + Cv0 [i_p-1, j + 1] +
Cv0 [i_p+ 1, j + 1]) + Cv0 [i_p, j-2] + Cv0 [i_p, j + 2] + Cv0 [i_p-2, j] +
Cv0 [i_p+ 2, j]) / 16 ・ ・ ・ Equation 27
Ch [i_p, j] = (4 ・ Ch0 [i_p, j] +2 ・ (Ch0 [i_p-1, j-1] + Ch0 [i_p+ 1, j-1] + Ch0 [i_p-1, j + 1] +
Ch0 [i_p+ 1, j + 1]) + Ch0 [i_p, j-2] + Ch0 [i_p, j + 2] + Ch0 [i_p-2, j] +
Ch0 [i_p+ 2, j]) / 16 ・ ・ ・ Formula 28
Note that << the peripheral addition method 1 >> corresponds to performing weighted addition of main similarity components between the interpolation target pixel and the peripheral pixels as shown in FIG. 4 (1), and << the peripheral addition method 2 >> This corresponds to performing weighted addition of main similarity components between the interpolation target pixel and the peripheral pixels as shown in FIG.
[0104]
By the way, the main similarity component calculated using the color information of the same color component, such as the above-mentioned main similarity component between GG, main similarity component between BB (RR), and main similarity component between RR (BB). (Hereinafter referred to as “same-color main similarity component”) is known to be suitable for evaluating similarity to an image having a low spatial frequency and a large number of colored portions. In addition, main similarity components calculated using color information of different color components, such as the main similarity component between GR (GB) and the main similarity component between BG (RG) It has been experimentally known that “component” is suitable for evaluating similarity to an image having a high spatial frequency and many achromatic portions. Furthermore, it has been experimentally known that the main similarity component between luminances is suitable for evaluating similarity to an image having a chromatic part and a part having a high spatial frequency to some extent.
[0105]
That is, the main similarity obtained by weighted addition of the main similarity component between the same colors, the main similarity component between different colors, and the main similarity component between luminances should be evaluated with high accuracy for various images. Is possible.
In addition, the three types of main similarity components calculated as the main similarity component between the same colors (the main similarity component between GG, the main similarity component between BB (RR), and the main similarity component between RR (BB)) are similar. The roles played in the evaluation of sex can be supplemented by color components, and two types of main similarity components (GR (GB) main similarity component, BG (RG) The main similarity component) can also complement each other in the role played in the similarity evaluation.
[0106]
Furthermore, in the first embodiment, the main similarity in the vertical direction is obtained by weighted addition of the result of the weighted addition of the main similarity component in the interpolation target pixel and the result of the weighted addition of the main similarity component in the surrounding pixels. Cv [i_p, j] and the horizontal similarity Ch [i_p, j] is calculated. Therefore, the major similarity Cv [i in the vertical direction_p, j] and the horizontal similarity Ch [i_p, j] can easily reflect the continuity of the color information between the interpolation target pixel and the peripheral pixels, and can determine wide-area similarity.
[0107]
In the first embodiment, the main similarity Cv [i in the vertical direction_p, j] and the horizontal similarity Ch [i_p, j] indicates that the smaller the value, the stronger the similarity.
As described above, the interpolation processing unit 17 performs the main similarity Cv [i in the vertical direction._p, j] and the horizontal similarity Ch [i_p, j], the major similarity Cv [i_p, j] and the horizontal similarity Ch [i_p, j], the similarity in the vertical direction and the horizontal direction (hereinafter referred to as “vertical and horizontal main similarity”) in the interpolation target pixel on the Bayer array is determined (S2 in FIG. 2). Then, as such a determination result, an index HV [i indicating the similarity in the vertical and horizontal directions_p, j] is set to the following value.
[0108]
For example, the interpolation processing unit 17 performs an arbitrary threshold Th1
| Cv [i_p, j] -Ch [i_p, j] | ＞ Th1 and Cv [i_p, j] <Ch [i_p, j]
Therefore, it is determined that the main similarity in the vertical direction is stronger than the horizontal direction, and the index HV [i_p, j] is set to 1 (S3 in FIG. 2),
| Cv [i_p, j] -Ch [i_p, j] | ＞ Th1 and Cv [i_p, j] ＞ Ch [i_p, j]
Is satisfied, it is determined that the main similarity in the horizontal direction is stronger than the vertical direction, and the index HV [i_p, j] is set to -1 (S4 in FIG. 2),
| Cv [i_p, j] -Ch [i_p, j] | ≦ Th1
If this holds, it is determined that the main similarity cannot be distinguished between the vertical and horizontal directions, and the index HV [i_p, j] is set to 0 (S5 in FIG. 2).
[0109]
Note that the threshold Th1 is the main similarity Cv [i in the vertical direction._p, j] and horizontal similarity Ch [i_p, j] is small, it serves to avoid misjudging that one main similarity is strong due to the influence of noise. Therefore, for a color image with a lot of noise, the accuracy of determining the main similarity can be improved by setting the threshold value Th1 high.
[0110]
Next, the interpolation processing unit 17 determines whether or not the direction in which the main similarity is strong is unknown in the determination of the vertical and horizontal main similarity as described above (S6 in FIG. 2), and the direction in which the main similarity is strong is unknown. In the vertical direction, the fine vertical portion similarity Cvs [i_p, j] "and the horizontal fine part similarity Chs [i_p, j] ”(S7 in FIG. 2).
[0111]
In the first embodiment, the direction in which the main similarity is strong is unknown is the index HV [i_p, j] is set to 0 (when the main similarity cannot be distinguished between the vertical and horizontal directions).
By the way, it is important to capture the sub-pixel pitch similarity that cannot be determined by the main similarity, and it is desirable that the fine part similarity be accurately calculated using as much color information as possible. The fine part similarity calculated as described below can determine the detailed similarity up to the sub-pixel pitch.
[0112]
Hereafter, the vertical fine part similarity Cvs [i_p, j] and the horizontal similarity Chs [i_p, j] will be described in detail.
First, the interpolation processing unit 17 calculates a plurality of types of fine part similarity components in the vertical direction and the horizontal direction defined by the following Expressions 29 to 34. Note that the coordinate display on the right side indicates the position on the sub-pixel type Bayer array in FIG.
[0113]
Fine part similarity component between RR (BB) in the vertical direction:
Cvs1 [i_p, j] = (| Z [i, j-2] -Z [i, j] | + | Z [i, j + 2] -Z [i, j] | +
| Z [i + 1, j-2] -Z [i + 1, j] | + | Z [i + 1, j + 2] -Z [i + 1, j] |) / 4 29
Fine part similarity component between horizontal RR (BB):
Chs1 [i_p, j] = | Z [i, j] -Z [i + 1, j] |
Fine part similarity component between GG in the vertical direction:
Cvs2 [i_p, j] = (| G [i, j-1] -G [i, j + 1] | + | G [i + 1, j-1] -G [i + 1, j + 1] |) / 2 ・ ・ ・ Formula 31
Lateral GG fine part similarity component:
Chs2 [i_p, j] = (| G [i, j-1] -G [i + 1, j-1] | + | G [i, j + 1] -G [i + 1, j + 1] |) / 2 ・ ・ ・ Formula 32
Fine part similarity component between vertical GR (GB):
Cvs3 [i_p, j] = (| G [i, j-1] -Z [i, j] | + | G [i, j + 1] -Z [i, j] | +
| G [i + 1, j-1] -Z [i + 1, j] | + | G [i + 1, j + 1] -Z [i + 1, j] |) / 4 33
Fine part similarity component between horizontal GR (GB):
Chs3 [i_p, j] = (| G [i-1, j] -Z [i, j] | + | G [i + 2, j] -Z [i + 1, j] |) / 2 Equation 34
In addition, although the element of the similarity which comprises each fine part similarity component mentioned above is calculated using the absolute value of a difference, you may calculate by the square of an absolute value, a power, etc.
[0114]
By the way, the distance between the color information used when calculating the fine part similarity component in the vertical direction is the same as the case of the main similarity component, but the fine part similarity component in the horizontal direction is the calculation of the main similarity component. Since it is a similarity component at subpixel intervals that sometimes disappears, it is possible to determine more detailed similarity.
FIG. 5 is a diagram showing the position of the color information used when calculating the fine part similarity component.
[0115]
By the way, as shown in FIG. 5, when the distance between two subpixels adjacent in the horizontal direction is L, the vertical RR fine portion similarity component and the vertical GG fine portion similarity component are 4L. It is calculated using the color information of the subpixels at a distant position, and the vertical GR fine portion similarity component is calculated using the color information of the subpixels at a position 2L away.
[0116]
Therefore, in the first embodiment, the interpolation processing unit 17 performs vertical addition when performing weighted addition of a plurality of types of fine part similarity components using weighting coefficients s1, s2, and s3 as shown in the following Expressions 35 and 36. The fine part similarity component between the horizontal direction and the horizontal direction is used after being normalized by the distance between the sub-pixels.
Cvs0 [i_p, j] = (s1 ・ Cvs1 [i_p, j] / 4 + s2 ・ Cvs2 [i_p, j] / 4 + s3 ・ Cvs3 [i_p, j] / 2) /
(s1 + s2 + s3) ... Formula 35
Chs0 [i_p, j] = (s1 ・ Chs1 [i_p, j] + s2 ・ Chs2 [i_p, j] + s3 ・ Chs3 [i_p, j]) /
(s1 + s2 + s3) ... Formula 36
In the first embodiment, normalization based on the distance between sub-pixels of the fine part similarity component is performed at the time of weighted addition of the fine part similarity component (at the time of calculation of Expression 35). By replacing the 29 denominator with 16, the 31 denominator with 8, and the 33 denominator with 8, the normalized fine part similarity component can be directly calculated (however, In this case, the operation of dividing each fine part similarity component in Expression 35 according to the distance is unnecessary).
[0117]
In Expressions 35 and 36, for example, “s1: s2: s3 = 1: 1: 1” can be considered as the ratio of the weighting coefficients s1, s2, and s3.
By the way, Cvs0 [i_p, j] and Chs0 [i_p, j] are left as they are, and the fine portion similarity Cvs [i_p, j] and the horizontal similarity Chs [i_p, j], here, an example of calculating these similarities by peripheral addition is shown.
[0118]
That is, the interpolation processing unit 17 obtains the result of the weighted addition for each direction of the fine part similarity component between the interpolation target pixel and the peripheral pixels (Cvs0 [i_p, j], Cvs0 [i_p-1, j-1], Cvs0 [i_p-1, j + 1], Cvs0 [i_p+ 1, j-1], Cvs0 [i_p+ 1, j + 1] etc.) is weighted and added as in the following << peripheral addition method 1 >> or << peripheral addition method 2 >>, and the fine portion similarity Cvs [i_p, j] and the horizontal similarity Chs [i_p, j] is calculated (corresponding to weighted addition of similarity components between the interpolation target pixel and the surrounding pixels as shown in FIG. 4).
[0119]
<< Peripheral addition method 1 >>
Cvs [i_p, j] = (4 ・ Cvs0 [i_p, j] + Cvs0 [i_p-1, j-1] + Cvs0 [i_p-1, j + 1]
+ Cvs0 [i_p+ 1, j-1] + Cvs0 [i_p+ 1, j + 1]) / 8 ・ ・ ・ Equation 37
Chs [i_p, j] = (4 ・ Chs0 [i_p, j] + Chs0 [i_p-1, j-1] + Chs0 [i_p-1, j + 1]
+ Chs0 [i_p+ 1, j-1] + Chs0 [i_p+ 1, j + 1]) / 8 ・ ・ ・ Equation 38
<< Peripheral addition method 2 >>
Cvs [i_p, j] = (4 ・ Cvs0 [i_p, j]
+2 ・ (Cvs0 [i_p-1, j-1] + Cvs0 [i_p+ 1, j-1] + Cvs0 [i_p-1, j + 1] + Cvs0 [i_p+ 1, j + 1])
+ Cvs0 [i_p, j-2] + Cvs0 [i_p, j + 2] + Cvs0 [i_p-2, j] + Cvs0 [i_p+ 2, j]) / 16 ・ ・ ・ Equation 39
Chs [i_p, j] = (4 ・ Chs0 [i_p, j]
+2 ・ (Chs0 [i_p-1, j-1] + Chs0 [i_p+ 1, j-1] + Chs0 [i_p-1, j + 1] + Chs0 [i_p+ 1, j + 1])
+ Chs0 [i_p, j-2] + Chs0 [i_p, j + 2] + Chs0 [i_p-2, j] + Chs0 [i_p+ 2, j]) / 16 ・ ・ ・ Formula 40
By the way, the fine part similarity component calculated using the color information of different color components (hereinafter, referred to as “different-color fine part similarity component”) like the above-described GR (GB) fine part similarity component. ) Is advantageous in determining the similarity in the fine part because the distance between the color information is the shortest in the vertical direction, but is suitable only for an image close to an achromatic color because the color components do not match. On the other hand, the fine part similarity component calculated using the color information of the same color component, such as the fine part similarity component between RR (BB) and the fine part similarity component between GG (hereinafter referred to as “micro part similarity between same colors”). The “degree component” is suitable for an image including a colored portion.
[0120]
In other words, the fine part similarity obtained by weighted addition of the different-color fine-part similarity component and the same-color fine-part similarity component enables high-precision evaluation of various images. is there.
[0121]
Further, in the first embodiment, the result of the weighted addition of the fine part similarity component in the interpolation target pixel and the result of the weighted addition of the fine part similarity component in the surrounding pixels are weighted and added, so that Similarity Cvs [i_p, j] and the horizontal similarity Chs [i_p, j] is calculated. Therefore, the vertical fine part similarity Cvs [i_p, j] and the horizontal similarity Chs [i_p, j] can easily reflect the continuity of the color information between the interpolation target pixel and the peripheral pixels, and can determine wide-area similarity.
[0122]
As described above, the interpolation processing unit 17 performs the fine portion similarity Cvs [i in the vertical direction._p, j] and the horizontal similarity Chs [i_p, j], the fine part similarity Cvs [i_p, j] and the horizontal similarity Chs [i_p, j], the similarity (hereinafter, referred to as “fine portion similarity”) of the sub-pixel pitch of the interpolation target pixel in the vertical direction and the horizontal direction is determined (FIG. 2 S8), and the vertical direction rather than the horizontal direction is determined. If it is determined that the fine part similarity is strong, the index HV [i_p, j] is replaced from 0 to 1 (S9 in FIG. 2).
[0123]
For example, the interpolation processing unit 17 determines the arbitrary threshold value Ths.
| Cvs [i_p, j] -Chs [i_p, j] | ＞ Ths and Cvs [i_p, j] <Chs [i_p, j]
Therefore, it is determined that the similarity in the vertical direction is stronger than the horizontal direction, and the index HV [i_p, j] is set to 1. The interpolation processing unit 17
| Cv [i_p, j] -Ch [i_p, j] | ≦ Ths
If this holds, it is determined that the fine part similarity is indistinguishable and the index HV [i_p, j] remain 0.
[0124]
By the way, since the subpixel divides the pixel in the vertical direction, even if interpolation processing is performed in units of subpixels, the resolution related to the similarity in the horizontal direction does not increase. In other words, regarding the similarity in the horizontal direction, the fine portion similarity does not obtain the amount of information greater than the main similarity.
Therefore, in the first embodiment,
| Cv [i_p, j] -Ch [i_p, j] | ＞ Ths and Cv [i_p, j] ＞ Ch [i_p, j]
The index HV [i_p, j] remain 0.
[0125]
Note that the threshold Ths plays a role in avoiding erroneous determination that one of the similarities is strong due to the influence of noise, as with the above-described threshold Th1.
Next, the interpolation processing unit 17 uses the index HV [i_p, j] is determined (S10 in FIG. 2), and green interpolation values G [i, j] and G [i + 1, j] of the two subpixels are calculated as follows.
[0126]

However, GvL, GvR, GhL, and GhR are values calculated as in the following << Average interpolation >> or << Color difference interpolation >>.
[0127]
《Average interpolation》
GvL = (G [i, j-1] + G [i, j + 1]) / 2 Equation 41
GvR = (G [i + 1, j-1] + G [i + 1, j + 1]) / 2 Equation 42
GhL = (2 ・ G [i-1, j] + G [i + 2, j]) / 3 Equation 43
GhR = (G [i-1, j] +2 ・ G [i + 2, j]) / 3 Equation 44
That is, in << average interpolation >>, GvL and GvR are average values of color information of subpixels in which a green component exists adjacent to each other in the vertical direction of each interpolation target subpixel. On the other hand, GhL and GhR are values obtained by averaging the color information of subpixels in which a green component is present in the horizontal direction of each interpolation target subpixel according to the distance from each interpolation target subpixel. .
[0128]
《Color difference interpolation》
GvL = (G [i, j-1] + G [i, j + 1]) / 2+ (2 ・ Z [i, j] -Z [i, j-2] -Z [i, j + 2 ]) / 4 ・ ・ ・ Formula 45
GvR = (G [i + 1, j-1] + G [i + 1, j + 1]) / 2
+ (2 · Z [i + 1, j] -Z [i + 1, j-2] -Z [i + 1, j + 2]) / 4 Equation 46
GhL = (2 ・ G [i-1, j] + G [i + 2, j]) / 3
+ (5 ・ Z [i, j] -2 ・ Z [i-3, j] -2 ・ Z [i + 1, j] -Z [i + 4, j]) / 9 Equation 47
GhR = (G [i-1, j] +2 ・ G [i + 2, j]) / 3
+ (5 ・ Z [i + 1, j] -2 ・ Z [i + 4, j] -2 ・ Z [i, j] -Z [i-3, j]) / 9 Equation 48
That is, << Color Difference Interpolation >> assumes that the color difference (corresponding to GR or GB) between the green component and the color component (red component or blue component) in the interpolation target subpixel is constant, and the vertical interpolation of the interpolation target subpixel. Interpolation of the color component (red component or blue component) of the subpixel in which the green component close to the direction (or horizontal direction) exists, calculation of the color difference component at that position, and the interpolation target using such a color difference This is equivalent to performing an operation for interpolating the color difference of the sub-pixel and converting the color difference of the sub-pixel to be interpolated into a green interpolation value all at once.
[0129]
Further, Expressions 45 to 48 are obtained by correcting the values calculated in the same manner as Expressions 41 to 44 of << average interpolation >> using color information of the same color as the interpolation target subpixel, thereby obtaining a green interpolation value. It can be said that this is an arithmetic expression for calculating. That is, in Equations 45 to 48, the first term is “local average information of the green component”, and the second term is the same as the “interpolation target subpixel” reflecting the unevenness information in the interpolation target subpixel. Local inflection information by the color component of ".
[0130]
However, when γ gradation conversion is performed on the image data before interpolation processing, γ conversion is close to logarithmic characteristics, so in << Color Difference Interpolation >>, the ratio of color components (G / R or G / B Will be assumed to be constant.
FIG. 6 is a diagram showing the position of color information used in << color difference interpolation >>.
As described above, the interpolation processing unit 17 determines the strength of the similarity at the pixel pitch based on the main similarity, and when the direction in which the similarity is strong is unknown by such determination, the fine portion similarity is determined. Is used to determine the strength of similarity at the sub-pixel pitch, but the main similarity and fine part similarity are calculated by weighted addition of the similarity component of the pixel to be interpolated and the similarity component of the surrounding pixels. The continuity of the color information between the interpolation target pixel and the peripheral pixels is easily reflected, and a wide range determination can be made. In addition, when calculating the fine part similarity, a plurality of types of fine part similarity components are weighted and added, so a lot of color information is used. At the time of such weighted addition, a plurality of types of fine part similarity are used. The components are normalized by the distance between subpixels.
[0131]
Therefore, in the G interpolation process of the first embodiment, it is possible to accurately determine the direction in which the similarity with the interpolation target pixel is strong, and therefore it is possible to calculate the interpolation value without breaking the harmony of the entire image. is there.
Next, the interpolation processing unit 17 performs RB interpolation processing.
Hereinafter, as the operations of the RB interpolation process, the operations of the two RB interpolation processes of the “RB interpolation process 1” and the “RB interpolation process 2” shown in FIG. 3 will be described. In the following description, two subpixels that should be supplemented with a red interpolation value (or blue interpolation value) are referred to as interpolation target subpixels. Further, since the blue interpolation value can be calculated in the same manner as the red interpolation value, description thereof will be omitted below.
[0132]
<< RB interpolation process 1 >>
First, in << RB interpolation process 1 >>, for each subpixel in which a missing color component exists in the interpolation target subpixel, the interpolation processing unit 17 calculates a green interpolation value (the G interpolation process described above) from the color information of the color component. The color difference is calculated (S10 in FIG. 3).
[0133]
For example, if the color information of the red component exists and the coordinates of two subpixels adjacent to each other are [i, j], [i + 1, j], the interpolation processing unit 17 sets the red component in these subpixels. The color difference Cr [i, j], Cr [i, j] including
Cr [i, j] = R [i, j] -G [i, j] Equation 49
Cr [i + 1, j] = R [i + 1, j] -G [i + 1, j] Equation 50
Calculated by
[0134]
Next, the interpolation processing unit 17 linearly interpolates the color difference of the interpolation target subpixel using the color difference calculated by the subpixel closest to the interpolation target subpixel (S11 in FIG. 3).
For example, assuming that the coordinates of two interpolation target subpixels are [m, n], [m + 1, n], and the color information of the red component is missing in these two interpolation target subpixels, interpolation is performed. When the color difference including the red component is calculated in the subpixels positioned in the vertical direction of the coordinates [m, n] and [m + 1, n] (corresponding to (1) in FIG. 7), the processing unit 17 Is the color difference Cr [m, n], Cr [m + 1, n] including the red component,
Cr [m, n] = (Cr [m, n-1] + Cr [m, n + 1]) / 2 Formula 51
Cr [m + 1, n] = (Cr [m + 1, n-1] + Cr [m + 1, n + 1]) / 2 Formula 52
Linear interpolation with
[0135]
Further, the interpolation processing unit 17 calculates a color difference including a red component in subpixels located in the horizontal direction of coordinates [m, n] and [m + 1, n] (corresponding to FIG. 7 (2)). Color difference including red component Cr [m, n], Cr [m + 1, n]
Cr [m, n] = (2 · Cr [m-1, n] + Cr [m + 2, n]) / 3 Formula 53
Cr [m + 1, n] = (Cr [m-1, n] + 2 · Cr [m + 2, n]) / 3 Equation 54
Linear interpolation with
[0136]
Further, the interpolation processing unit 17 calculates the color difference including the red component in the subpixels located in the oblique direction of the coordinates [m, n] and [m + 1, n] (corresponding to FIG. 7 (3)). The color difference Cr [m, n], Cr [m + 1, n] including the red component,
Cr [m, n] = (2 ・ (Cr [m-1, n-1] + Cr [m-1, n + 1])
+ Cr [m + 2, n-1] + Cr [m + 2, n + 1]) / 6 Equation 55
Cr [m + 1, n] = (Cr [m-1, n-1] + Cr [m-1, n + 1]
+2 ・ (Cr [m + 2, n-1] + Cr [m + 2, n + 1])) / 6 ... Formula 56
Linear interpolation with
[0137]
Next, the interpolation processing unit 17 converts the color difference into an interpolation value for each sub-pixel to be interpolated using color information of the green component (which may be a green interpolation value) (S12 in FIG. 3). That is, the interpolation processing unit 17 performs the red interpolation values R [m, n], R [m + 1, n] of the subpixels located at the arbitrary coordinates [m, n], [m + 1, n] described above. ]about,
R [m, n] = Cr [m, n] + G [m, n] Equation 57
R [m + 1, n] = Cr [m + 1, n] + G [m + 1, n] Equation 58
Calculated by
[0138]
By the way, << RB interpolation process 1 >> has a problem that a false color remains in the vicinity of an isolated point (image portion having a high spatial frequency with low similarity to surrounding pixels). Therefore, << RB interpolation process 1 >> is performed. Later, post processing may be performed in which median filters are individually applied to the hue a and b planes obtained by converting the image data into the Lab color system.
[0139]
Further, when the median filter is applied in this way, since the filter size of 3 × 3 (= 9 points) on the Bayer array hardly works, the filter size is 5 × 5 (= 25 points) on the Bayer array. It is necessary to set in a wide area.
That is, in order to obtain red and blue interpolation values from which the false color around the isolated point is removed by << RB interpolation process 1 >>, it is necessary to separately perform median processing with a filter size set to a wide range. Heavy processing will be imposed. Also, increasing the filter size in the median process leads to an increased risk of destroying the fine structure of the chromatic part (hereinafter referred to as “color structure”).
[0140]
Therefore, in the following, an RB interpolation process capable of calculating red and blue interpolation values at high speed and high accuracy without leaving a false color around an isolated point will be presented as << RB interpolation process 2 >>. Note that << RB interpolation process 2 >> corresponds to claims 53 to 58.
<< RB interpolation process 2 >>
First, in << RB interpolation process 2 >>, the interpolation processing unit 17 calculates a color difference for each sub-pixel in which a missing color component exists in the interpolation target sub-pixel, as in << RB interpolation process 1 >> (S20 in FIG. 3). .
[0141]
Note that, in the state where the color difference including the red component is calculated by such processing, the color difference including the red component is oblique to two subpixels in which the color information of the red component is missing and the color information of the blue component is present. It will be arranged so as to surround from four directions.
The interpolation processing unit 17 interpolates the color difference of the interpolation target sub-pixel with the median value of the color difference of the sub-pixels positioned in the diagonal direction for each of the interpolation target sub-pixels surrounded by the four sub-pixels with the calculated sub-pixel. FIG. 3S21).
[0142]
For example, as shown in FIG. 8 (1), the coordinates of any two subpixels surrounded by four diagonal directions by subpixels for which the color difference Cr including the red component is calculated are represented by [m, n], [m + 1, n], the interpolation processing unit 17 calculates the color difference Cr [m, n], Cr [m + 1, n] including red in these two subpixels,
Cr [m, n] = median {Cr [m-1, n-1], Cr [m-1, n + 1],
Cr [m + 2, n-1], Cr [m + 2, n + 1]} Equation 59
Cr [m + 1, n] = median {Cr [m-1, n-1], Cr [m-1, n + 1],
Cr [m + 2, n-1], Cr [m + 2, n + 1]} ... Formula 60
Calculated by However, median {} indicates a function for calculating median values of a plurality of elements, and when the number of elements is an even number, an average value of the middle two elements is assumed.
[0143]
In the state where the color difference including the red component is calculated as described above, the color difference including the red component surrounds the two sub-pixels in which the color information of the red component and the blue component is missing from the four directions of up, down, left, and right. Will be placed.
The interpolation processing unit 17 interpolates the color difference of the interpolation target subpixel by the median value of the color difference of the subpixels positioned in the vertical and horizontal directions for each of the interpolation target subpixels surrounded by the subpixels in which the color difference is calculated. (S22 in FIG. 3).
[0144]
For example, as shown in FIG. 8 (2), the coordinates of any two sub-pixels surrounded by four sub-pixels in which the color difference Cr including the red component is calculated are represented by [m, n], [m Assuming that + 1, n], the interpolation processing unit 17 calculates the color difference Cr [m, n], Cr [m + 1, n] including red in these two subpixels,
Cr [m, n] = median {Cr [m-1, n], Cr [m + 2, n],
Cr [m, n-1], Cr [m, n + 1]} Equation 61
Cr [m + 1, n] = median {Cr [m-1, n], Cr [m + 2, n],
Cr [m + 1, n-1], Cr [m + 1, n + 1]} Equation 62
Calculated by
[0145]
Next, the interpolation processing unit 17 converts the color difference into an interpolated value based on the color component color information for each sub-pixel to be interpolated, similarly to the << RB interpolation process 1 >> (S23 in FIG. 3).
As described above, in << RB interpolation process 2 >>, the median process is performed only for the color difference representing the hue and not for the luminance component. Further, in the RB interpolation process, when the red interpolation value is supplemented to the two subpixels marked with a circle in FIG. 9A, the color difference Cr located in the 3 × 5 range on the Bayer array is used. Since the color difference including the red component in the sub-pixel to which the mark is given is calculated, the color difference including the red component in the two sub-pixels to which the mark is given is the color difference located in the 3 × 5 range on the Bayer array A value close to the result of median processing with weighting Cr. Further, when the red interpolation value is supplemented to the two subpixels to which the Δ mark in FIG. 9 (2) is added, the X mark is given by using the color difference Cr located in the 5 × 3 range on the Bayer array. Since the color difference including the red component in the sub-pixel is calculated, the color difference including the red component in the two sub-pixels to which the Δ mark is given weights the color difference Cr located in the 5 × 3 range on the Bayer array. Value close to the result of median processing.
[0146]
That is, << RB Interpolation Process 2 >> has substantially the same effect as a wide area median process while keeping the filter size small, so that the false color around the isolated point can be obtained without destroying the color structure. Generation can be reduced.
Further, in << RB interpolation process 2 >>, the median process is performed while performing the interpolation process. Therefore, the processing efficiency is higher than that in the case where the median process is performed after the << RB interpolation process 1 >> is performed, and FIG. In each median process with 3S22, since only four color differences are used, a very high-speed median process is possible.
[0147]
Therefore, << RB interpolation process 2 >> can calculate interpolation values at high speed while reducing the occurrence of false colors around isolated points without destroying the color structure.
By the way, in the G interpolation process in the first embodiment described above, a false color is likely to occur at a color boundary portion where the color difference is not constant.
Further, when the chromatic aberration of magnification exists in the photographing optical system 12, the red component and the blue component are imaged at positions slightly shifted from the image forming position of the green component as shown in FIG. Such a phenomenon is called “color shift”.) Therefore, in the first embodiment, the color difference cannot be appropriately calculated due to such a color shift or the like, and a false color may occur. In particular, in the G interpolation process to which << color difference interpolation >> is applied, there is a possibility that a false color is generated due to overcorrection due to lateral chromatic aberration described later.
[0148]
FIG. 11 is a diagram for explaining overcorrection due to magnification aberration.
In FIG. 11, when the chromatic aberration of magnification does not exist in the photographing optical system 12 and the color information of each sub-pixel is given as indicated by the mark ● in FIG. 11 (1) (the image data to be interpolated) If the similarity is strong in the horizontal direction, the color information of the green component is constant, and the color information of the red component and blue component changes gently in the vicinity of the interpolation target subpixel (subpixel where Z5 and Z6 exist)), Since the second term of Equation 47 and Equation 48 is 0, the value obtained by averaging G4 and G7 (corresponding to the first term of Equation 47 and Equation 48) is corrected without being corrected to G5 or G6 (green Interpolation value).
[0149]
However, when Z1, Z2, Z5, Z6, Z9, and Z10 are red component color information, and the color information of these red components is shifted by 2 sub-pixels in the right direction due to the chromatic aberration of magnification of the photographing optical system 12. The color information of each sub-pixel changes as shown in FIG. Therefore, in such a case, the second term of Equation 47 or Equation 48 does not become 0, and the value obtained by averaging G4 and G7 is excessively corrected (hereinafter referred to as “overcorrection”). G5 and G6, which should be the same value as G7 and G7, are larger than G4 and G7.
[0150]
When Z1, Z2, Z5, Z6, Z9, and Z10 are blue component color information, and the blue component color information is shifted by 2 subpixels in the left direction due to the chromatic aberration of magnification of the photographing optical system 12. Originally, G5 and G6, which should be similar to G4 and G7, are smaller than G4 and G7.
Therefore, in the second embodiment, an interpolation process capable of suppressing the occurrence of false colors due to such overcorrection is presented.
[0151]
<< Second Embodiment >>
The operation of the second embodiment will be described below. Note that the RB interpolation process in the second embodiment can be performed in the same manner as in the first embodiment, and thus the description thereof is omitted here.
12 and 13 are operation flowcharts of the interpolation processing unit 17 in the second embodiment, and show the operation of the interpolation processing unit 17 in the G interpolation processing.
[0152]
Hereinafter, although the G interpolation process in the second embodiment will be described, detailed description of the same operations as those in the first embodiment will be omitted.
First, as in the first embodiment, the interpolation processing unit 17 performs the main similarity Cv [i in the vertical direction._p, j] and horizontal similarity Ch [i_p, j] (FIG. 12S1), the vertical and horizontal main similarities are compared (FIG. 12S2), and the index HV [i_p, j] is set (FIG. 12 S3 to S5).
[0153]
Next, the interpolation processing unit 17 performs the main similarity C45 [i] in the oblique 45 degree direction on the interpolation target pixel on the Bayer array._p, j] and the main similarity C135 [i_p, j] is calculated (S6 in FIG. 12).
Here, the degree of similarity C45 [i_p, j] and the similarity C135 [i_p, j] will be described in detail.
[0154]
First, the interpolation processing unit 17 calculates a plurality of types of main similarity components for the oblique 45 degree direction and the oblique 135 degree direction defined by the following Expressions 63 to 70.
Major similarity components between GGs in a 45-degree direction:
C45_1 [i_p, j] = (| G [i_p, j-1] -G [i_p-1, j] | + | G [i_p+ 1, j] -G [i_p, j + 1] |) / 2 ・ ・ ・ Equation 63
Major similarity component between GG in the direction of diagonal 135 degrees:
C135_1 [i_p, j] = (| G [i_p, j-1] -G [i_p+ 1, j] | + | G [i_p-1, j] -G [i_p, j + 1] |) / 2 ... Formula 64
Major similarity component between BB (RR) in the 45 ° direction:
C45_2 [i_p, j] = | Z [i_p+ 1, j-1] -Z [i_p-1, j + 1] |
Main similarity component between BB (RR) in the direction of diagonal 135 degrees:
C135_2 [i_p, j] = | Z [i_p-1, j-1] -Z [i_p+ 1, j + 1] |
Major similarity component between RR (BB) in the direction of 45 degrees diagonally:
C45_3 [i_p, j] = (| Z [i_p+ 2, j-2] -Z [i_p, j] | + | Z [i_p-2, j + 2] -Z [i_p, j] |) / 2 ・ ・ ・ Equation 67
Major similarity component between RR (BB) in the direction of diagonal 135 degrees:
C135_3 [i_p, j] = (| Z [i_p-2, j-2] -Z [i_p, j] | + | Z [i_p+ 2, j + 2] -Z [i_p, j] |) / 2 ・ ・ ・ Formula 68
Major similarity component between BR (RB) in the 45 ° direction:
C45_4 [i_p, j] = (| Z [i_p+ 1, j-1] -Z [i_p, j] | + | Z [i_p-1, j + 1] -Z [i_p, j] |) / 2 ・ ・ ・ Equation 69
Major similarity component between BR (RB) in diagonal direction of 135 degrees:
C135_4 [i_p, j] = (| Z [i_p-1, j-1] -Z [i_p, j] | + | Z [i_p+ 1, j + 1] -Z [i_p, j] |) / 2 ・ ・ ・ Formula 70
In addition, although the element of the similarity which comprises each main similarity component mentioned above is calculated using the absolute value of a difference, you may calculate by the square of an absolute value, a power, etc.
[0155]
Next, the interpolation processing unit 17 performs weighted addition of a plurality of types of main similarity components for each direction using weighting coefficients b1, b2, b3, and b4 as shown in the following Expression 71 and Expression 72.
C45_0 [i_p, j] = (b1, C45_1 [i_p, j] + b2 ・ C45_2 [i_p, j] + b3 ・ C45_3 [i_p, j]
+ b4 ・ C45_4 [i_p, j]) / (b1 + b2 + b3 + b4) ... Equation 71
C135_0 [i_p, j] = (b1, C135_1 [i_p, j] + b2 ・ C135_2 [i_p, j] + b3 ・ C135_3 [i_p, j]
+ b4 ・ C135_4 [i_p, j]) / (b1 + b2 + b3 + b4) ... Formula 72
In Formula 71 and Formula 72, for example, “b1: b2: b3: b4 = 2: 1: 1: 2” may be considered as the ratio of the weighting coefficients b1, b2, b3, and b4.
[0156]
C45_0 [i calculated in this way_p, j] and C135_0 [i_p, j] is the main similarity C45 [i] in the 45 ° oblique direction of the pixel to be interpolated as it is._p, j] and the main similarity C135 [i_p, j], here, an example of calculating these similarities by peripheral addition is shown.
That is, the interpolation processing unit 17 performs the result of weighted addition for each direction of the main similarity component between the interpolation target pixel and the peripheral pixels (C45 — 0 [i_p, j], C45_0 [i_p-1, j-1], C45_0 [i_p-1, j + 1], C45_0 [i_p+ 1, j-1], C45_0 [i_p+ 1, j + 1], etc.) are weighted and added as in the following << peripheral addition method 1 >> or << peripheral addition method 2 >>, and the main similarity C45 [i_p, j] and the main similarity C135 [i_p, j] is calculated (corresponding to weighted addition of similarity components between the interpolation target pixel and the surrounding pixels as shown in FIG. 4).
[0157]
<< Peripheral addition method 1 >>
C45 [i_p, j] = (4 ・ C45_0 [i_p, j] + C45_0 [i_p-1, j-1] + C45_0 [i_p+ 1, j-1]
+ C45_0 [i_p-1, j + 1] + C45_0 [i_p+ 1, j + 1]) / 8 ・ ・ ・ Formula 73
C135 [i_p, j] = (4 ・ C135_0 [i_p, j] + C135_0 [i_p-1, j-1] + C135_0 [i_p+ 1, j-1]
+ C135_0 [i_p-1, j + 1] + C135_0 [i_p+ 1, j + 1]) / 8 ・ ・ ・ Formula 74
<< Peripheral addition method 2 >>
C45 [i_p, j] = (4 ・ C45_0 [i_p, j] +2 ・ (C45_0 [i_p-1, j-1] + C45_0 [i_p+ 1, j-1]
+ C45_0 [i_p-1, j + 1] + C45_0 [i_p+ 1, j + 1]) + C45_0 [i_p, j-2]
+ C45_0 [i_p, j + 2] + C45_0 [i_p-2, j] + C45_0 [i_p+ 2, j]) / 16 ・ ・ ・ Formula 75
C135 [i_p, j] = (4 ・ C135_0 [i_p, j] +2 ・ (C135_0 [i_p-1, j-1] + C135_0 [i_p+ 1, j-1]
+ C135_0 [i_p-1, j + 1] + C135_0 [i_p+ 1, j + 1]) + C135_0 [i_p, j-2]
+ C135_0 [i_p, j + 2] + C135_0 [i_p-2, j] + C135_0 [i_p+ 2, j]) / 16 ・ ・ ・ Equation 76
The main similarity C45 [i in the 45 ° oblique direction calculated in this way_p, j] and the main similarity C135 [i_p, j], the weighted addition with a plurality of types of main similarity components and peripheral pixels is performed using the above-described main similarity Cv [i_p, j] and the horizontal similarity Ch [i_p, j]. In the second embodiment, the main similarity C45 [i_p, j] and the main similarity C135 [i_p, j] indicates that the smaller the value, the stronger the similarity.
[0158]
The interpolation processing unit 17 calculates the main similarity C45 [i_p, j] and the main similarity C135 [i_p, j], the main similarity C45 [i_p, j] and the main similarity C135 [i_p, j], the similarity (hereinafter, referred to as “oblique main similarity”) of the interpolation target pixel with respect to the oblique 45 ° direction and the oblique 135 ° direction is determined (S7 in FIG. 12). Then, as such a determination result, an index DN [i indicating the similarity in the diagonal direction_p, j] is set to the following value.
[0159]
For example, the interpolation processing unit 17 performs an arbitrary threshold Th2
| C45 [i_p, j] -C135 [i_p, j] | ＞ Th2 and C45 [i_p, j] <C135 [i_p, j]
Therefore, it is determined that the main similarity in the 45 degree direction is stronger than the 135 degree direction, and the indicator DN [i_p, j] is set to 1 (S8 in FIG. 12),
| C45 [i_p, j] -C135 [i_p, j] | ＞ Th2 and C45 [i_p, j] ＞ C135 [i_p, j]
Therefore, it is determined that the main similarity in the oblique 135 degree direction is stronger than the oblique 45 degree direction, and the indicator DN [i_p, j] is set to -1 (S9 in FIG. 12),
| C45 [i_p, j] -C135 [i_p, j] | ≦ Th2
If it holds, the indicator DN [i_p, j] is set to 0 (S10 in FIG. 12).
[0160]
Note that the threshold value Th2 plays a role of avoiding erroneous determination that one of the similarities is strong due to the influence of noise, like the threshold value Th1 used for the comparison of vertical and horizontal main similarities in the first embodiment.
Next, the interpolation processing unit 17 determines whether or not the direction in which the main similarity is strong is unknown in the determination of the vertical and horizontal main similarities and the diagonal main similarities as described above (S11 in FIG. 12). When the strong direction is unknown, as in the first embodiment, the vertical fine portion similarity Cvs [i_p, j] and the horizontal similarity Chs [i_p, j] is calculated (S12 in FIG. 12).
[0161]
In the second embodiment, the direction in which the main similarity is strong is unknown is the index HV [i_p, j] is set to 0 and the index DN [i_p, j] is set to 0 (when the main similarity cannot be distinguished between the vertical and horizontal directions, and the diagonal main similarity cannot be distinguished).
The interpolation processing unit 17 calculates the fine portion similarity Cvs [i in the vertical direction._p, j] and the horizontal similarity Chs [i_p, j] is calculated, as in the first embodiment, the strength of the fine part similarity is determined (S13 in FIG. 12), and when it is determined that the fine part similarity in the vertical direction is stronger than the horizontal direction, Indicator HV [i_p, j] is replaced with 1 (S14 in FIG. 12).
[0162]
Next, the interpolation processing unit 17 uses an index HV [i indicating the similarity in the vertical and horizontal directions._p, j] and the index DN [i_p, j] is determined (S15 in FIG. 12), and the similarity strength of the interpolation target pixel is classified into one of the following cases 1 to case 9.
case1: (HV [i_p, j], DN [i_p, j]) = (1,1): The similarity in the vertical and diagonal 45 degrees directions is strong.
case2: (HV [i_p, j], DN [i_p, j]) = (1,0): Similar in the vertical direction.
[0163]
case3: (HV [i_p, j], DN [i_p, j]) = (1, -1): Strong similarity in the vertical and diagonal 135 degrees directions.
case4: (HV [i_p, j], DN [i_p, j]) = (0,1): Strong similarity in the 45 ° direction.
case5: (HV [i_p, j], DN [i_p, j]) = (0,0): The similarity in all directions is strong or the similarity in all directions is weak.
case6: (HV [i_p, j], DN [i_p, j]) = (0, -1): Strong similarity in the direction of oblique 135 degrees.
[0164]
case7: (HV [i_p, j], DN [i_p, j]) = (-1,1): The similarity in the horizontal and oblique 45 degree directions is strong.
case8: (HV [i_p, j], DN [i_p, j]) = (-1,0): Similar in horizontal direction.
case9: (HV [i_p, j], DN [i_p, j]) = (-1, -1): Strong similarity in the horizontal and oblique 135 degree directions.
FIG. 14 shows (HV [i_p, j], DN [i_p, j]) is a diagram showing a direction of strong similarity corresponding to the value.
[0165]
By the way, in FIG. 14, “case5: (HV [i_p, j], DN [i_p, j]) = (0,0) ”is not displayed, but as in case 5, the similarity in all directions is strong or the similarity in all directions is weak. This corresponds to the target pixel belonging to a flat portion or an isolated point (an image portion having a high spatial frequency with low similarity to surrounding pixels).
[0166]
Next, the interpolation processing unit 17 calculates green interpolation values G [i, j], G [i + 1, j] as follows according to the determination result described above.

However,
GvL = (G [i, j-1] + G [i, j + 1]) / 2
+ (2 ・ Z [i, j] -Z [i, j-2] -Z [i, j + 2]) / 8
+ (2 ・ (2 ・ G [i-1, j] -G [i-1, j-2] -G [i-1, j + 2])
+2 ・ G [i + 2, j] -G [i + 2, j-2] -G [i + 2, j + 2]) / 24
Gv45L = (G [i, j-1] + G [i, j + 1]) / 2
+ (2 ・ Z [i, j] -Z [i, j-2] -Z [i, j + 2]) / 8
+ (2 ・ (2 ・ Z [i-1, j + 1] -Z [i-1, j-1] -Z [i-1, j + 3])
+2 ・ Z [i + 2, j-1] -Z [i + 2, j-3] -Z [i + 2, j + 1]) / 24 ・ ・ ・ Formula 78
Gv135L = (G [i, j-1] + G [i, j + 1]) / 2
+ (2 ・ Z [i, j] -Z [i, j-2] -Z [i, j + 2]) / 8
+ (2 ・ (2 ・ Z [i-1, j-1] -Z [i-1, j-3] -Z [i-1, j + 1])
+2 ・ Z [i + 2, j + 1] -Z [i + 2, j-1] -Z [i + 2, j + 3]) / 24
GhL = (2 ・ G [i-1, j] + G [i + 2, j]) / 3
+ (5 ・ Z [i, j] -2 ・ Z [i-3, j] -2 ・ Z [i + 1, j] -Z [i + 4, j]) / 18
+ (5 ・ G [i, j-1] -2 ・ G [i-3, j-1] -2 ・ G [i + 1, j-1] -G [i + 4, j-1]
+5 ・ G [i, j + 1] -2 ・ G [i-3, j + 1] -2 ・ G [i + 1, j + 1] -G [i + 4, j + 1]) / 36 ・ ・ ・ Formula 80
Gh45L = (2 ・ G [i-1, j] + G [i + 2, j]) / 3
+ (5 ・ Z [i, j] -2 ・ Z [i-3, j] -2 ・ Z [i + 1, j] -Z [i + 4, j]) / 18
+ (5 ・ Z [i + 2, j-1] -2 ・ Z [i-1, j-1] -2 ・ Z [i + 3, j-1] -Z [i + 6, j-1 ]
+5 ・ Z [i-2, j + 1] -2 ・ Z [i-1, j + 1] -2 ・ Z [i-5, j + 1] -Z [i + 2, j + 1] ) / 36 ・ ・ ・ Formula 81
Gh135L = (2 ・ G [i-1, j] + G [i + 2, j]) / 3
+ (5 ・ Z [i, j] -2 ・ Z [i-3, j] -2 ・ Z [i + 1, j] -Z [i + 4, j]) / 18
+ (5 ・ Z [i-2, j-1] -2 ・ Z [i-5, j-1] -2 ・ Z [i-1, j-1] -Z [i + 2, j-1 ]
+5 ・ Z [i + 2, j + 1] -2 ・ Z [i-1, j + 1] -2 ・ Z [i + 3, j + 1] -Z [i + 6, j + 1] ) / 36 ・ ・ ・ Formula 82
GvR = (G [i + 1, j-1] + G [i + 1, j + 1]) / 2
+ (2 ・ Z [i + 1, j] -Z [i + 1, j-2] -Z [i + 1, j + 2]) / 8
+ (2 ・ G [i-1, j] -G [i-1, j-2] -G [i-1, j + 2]
+2 ・ (2 ・ G [i + 2, j] -G [i + 2, j-2] -G [i + 2, j + 2])) / 24 ・ ・ ・ Equation 83
Gv45R = (G [i + 1, j-1] + G [i + 1, j + 1]) / 2
+ (2 ・ Z [i + 1, j] -Z [i + 1, j-2] -Z [i + 1, j + 2]) / 8
+ (2 ・ Z [i-1, j + 1] -Z [i-1, j-1] -Z [i-1, j + 3]
+2 ・ (2 ・ Z [i + 2, j-1] -Z [i + 2, j-3] -Z [i + 2, j + 1])) / 24 ・ ・ ・ Equation 84
Gv135R = (G [i + 1, j-1] + G [i + 1, j + 1]) / 2
+ (2 ・ Z [i + 1, j] -Z [i + 1, j-2] -Z [i + 1, j + 2]) / 8
+ (2 ・ Z [i-1, j-1] -Z [i-1, j-3] -Z [i-1, j + 1]
+2 ・ (2 ・ Z [i + 2, j + 1] -Z [i + 2, j-1] -Z [i + 2, j + 3])) / 24 ・ ・ ・ Equation 85
GhR = (G [i-1, j] +2 ・ G [i + 2, j]) / 3
+ (5 ・ Z [i + 1, j] -2 ・ Z [i + 4, j] -2 ・ Z [i, j] -Z [i-3, j]) / 18
+ (5 ・ G [i + 1, j-1] -2 ・ G [i + 4, j-1] -2 ・ G [i, j-1] -G [i-3, j-1]
+5 ・ G [i + 1, j + 1] -2 ・ G [i + 4, j + 1] -2 ・ G [i, j + 1] -G [i-3, j + 1]) / 36 ・ ・ ・ Formula 86
Gh45R = (G [i-1, j] +2 ・ G [i + 2, j]) / 3
+ (5 ・ Z [i + 1, j] -2 ・ Z [i + 4, j] -2 ・ Z [i, j] -Z [i-3, j]) / 18
+ (5 ・ Z [i + 3, j-1] -2 ・ Z [i + 6, j-1] -2 ・ Z [i + 2, j-1] -Z [i-1, j-1 ]
+5 ・ Z [i-1, j + 1] -2 ・ Z [i + 2, j + 1] -2 ・ Z [i-2, j + 1] -Z [i-5, j + 1] ) / 36 ・ ・ ・ Formula 87
Gh135R = (G [i-1, j] +2 ・ G [i + 2, j]) / 3
+ (5 ・ Z [i + 1, j] -2 ・ Z [i + 4, j] -2 ・ Z [i, j] -Z [i-3, j]) / 18
+ (5 ・ Z [i-1, j-1] -2 ・ Z [i + 2, j-1] -2 ・ Z [i-2, j-1] -Z [i-5, j-1 ]
+5 ・ Z [i + 3, j + 1] -2 ・ Z [i + 6, j + 1] -2 ・ Z [i + 2, j + 1] -Z [i-1, j + 1] ) / 36 ・ ・ ・ Formula 88
It is.
[0167]
15 to 17 are diagrams illustrating the position of color information used when calculating the green interpolation values G [i, j] and G [i + 1, j].
By the way, in the formulas 77 to 88, the first term is “local average information of the green component”, and the second term is “local inflection information by the same color component as that of the interpolation target subpixel”. The third term is “local inflection information with a color component different from that of the interpolation target sub-pixel”.
[0168]
Here, “local inflection information by the same color component as the interpolation target pixel” of Gv45L and Gv45R is local inflection information having directionality in the vertical direction, and “color component different from the interpolation target pixel”. `` Local inflection information by '' is local inflection information having directionality in the vertical direction and 45 degrees oblique direction, Gv135L, Gv135R `` Local inflection due to the same color component as the interpolation target pixel '' “Information” is local inflection information having directionality in the vertical direction, and “local inflection information by a color component different from that of the interpolation target pixel” has directionality in the vertical direction and an oblique 135 degree direction. Gh45L and Gh45R “local inflection information with the same color component as the interpolation target pixel” is local inflection information with directionality in the horizontal direction. `` Local inflection information with color components different from the target pixel '' is Gh135L, Gh135R “local inflection information with the same color component as the interpolation target pixel” is local inflection information with directionality in the horizontal direction. “Local inflection information by a color component different from that of the interpolation target pixel” is local inflection information having directionality in the horizontal direction and the oblique 135 degree direction.
[0169]
Also, GvL and GvR “local inflection information with the same color component as the interpolation target pixel” and “local inflection information with a color component different from the interpolation target pixel” both have a vertical directionality. GhL, GhR "local inflection information with the same color component as the interpolation target pixel" and "local inflection information with a color component different from the interpolation target pixel" , Both are local inflection information having directionality in the lateral direction.
[0170]
As described above, since the interpolation processing unit 17 calculates the interpolation value using the inflection information of a plurality of color components, the second embodiment is a case where the chromatic aberration of magnification exists in the photographing optical system 12. However, it is possible to appropriately correct the average information, and the generation of false colors can be suppressed. This is shown in FIG.
Further, in the second embodiment, the inflection information of a plurality of color components cancel each other, so that it is possible to suppress false color at a color boundary portion where the color difference is not constant.
[0171]
That is, in the G interpolation process of the second embodiment, it is possible to suppress false colors that are highly likely to occur due to the G interpolation process of the first embodiment.
By the way, the inflection information of a plurality of color components is likely to change in value depending on which direction of color information is used. Therefore, when calculating the interpolation value using the inflection information of a plurality of color components, the inflection information that accurately determines the direction of strong similarity in the interpolation target pixel and adopts the color information located in that direction. Must be used. In the second embodiment, by using the similarity component in the diagonal direction in addition to the similarity component in the vertical direction and the horizontal direction, the accuracy of determining the strength of similarity in the interpolation target pixel is improved.
[0172]
<< Third Embodiment >>
The operation of the third embodiment will be described below. Note that the RB interpolation processing in the third embodiment can be performed in the same manner as in the first embodiment, and thus description thereof is omitted here.
FIGS. 19 to 21 are operation flowcharts of the interpolation processing unit 17 in the third embodiment, and show operations of the interpolation processing unit 17 in the G interpolation processing.
[0173]
Note that the difference in the G interpolation processing between the third embodiment and the first embodiment is that the similarity in the vertical and horizontal directions is determined and the index HV [i_p, j] is different, and the third embodiment performs the processes of FIGS. 19S1 to S4, FIGS. 20 and 21 instead of the processes of FIGS. 2S1 to S9 of the first embodiment. It corresponds to that.
In the third embodiment, the index HV [i_p, j] in the process of setting the interpolation target pixel [i_p, j] in the vertical direction and the horizontal direction, the main similarity and the fine part similarity are calculated separately for the different color similarity and the same color similarity, respectively. Therefore, here, in order to simplify the description of the G interpolation process described later, a process of calculating each similarity performed by the interpolation processing unit 17 will be described.
[0174]
First, the similarity between different colors of the main similarity in the vertical direction and the horizontal direction (hereinafter referred to as “main similarity between main colors”) and the similarity between the same colors (hereinafter referred to as “main similar similarity between colors”). A process of calculating?
The interpolation processing unit 17 performs a plurality of types of main similarity components (Cv1 [i] for the vertical direction and the horizontal direction defined by Expressions 10 to 19 of the first embodiment._p, j], Ch1 [i_p, j], Cv2 [i_p, j], Ch2 [i_p, j], Cv3 [i_p, j], Ch3 [i_p, j], Cv4 [i_p, j], Ch4 [i_p, j], Cv5 [i_p, j], Ch5 [i_p, j]).
[0175]
Next, the interpolation processing unit 17 performs weighted addition of a plurality of types of similarity components between the same colors by weighting coefficients a1, a2, and a3 as shown in the following Expressions 89 and 90, and Expression 91 and As shown in Equation 92, weighting coefficients a4 and a5 are used to weight-add multiple types of different-color similarity components for each direction.
CvD0 [i_p, j] = (a1 ・ Cv1 [i_p, j] + a2 ・ Cv2 [i_p, j] + a3 ・ Cv3 [i_p, j])
/(a1+a2+a3)...Formula 89
ChD0 [i_p, j] = (a1 ・ Ch1 [i_p, j] + a2 ・ Ch2 [i_p, j] + a3 ・ Ch3 [i_p, j])
/(a1+a2+a3)...Formula 90
CvN0 [i_p, j] = (a4 ・ Cv4 [i_p, j] + a5 ・ Cv5 [i_p, j]) / (a4 + a5) ... 91
ChN0 [i_p, j] = (a4 ・ Ch4 [i_p, j] + a5 ・ Ch5 [i_p, j]) / (a4 + a5) ...
CvD0 [i calculated in this way_p, j], ChD0 [i_p, j], CvN0 [i_p, j], ChN0 [i_p, j] are used as they are, and the similarity CvD [i_p, j], horizontal similarity between similar colors ChD [i_p, j], similarity between major different colors in the vertical direction CvN [i_p, j], horizontal similarity between different colors ChN [i_p, j], here, an example of calculating these similarities by peripheral addition is shown.
[0176]
That is, the interpolation processing unit 17 obtains a value (CvD0 [i] obtained by weighted addition of the similarity components of the interpolation target pixel and the peripheral pixels._p, j], CvD0 [i_p-1, j-1], CvD0 [i_p-1, j + 1], CvD0 [i_p+ 1, j-1], CvD0 [i_p+ 1, j + 1], etc.) in consideration of the continuity of similarity, weighted addition is performed as in the following << peripheral addition method 1 >> or << peripheral addition method 2 >> Similar similarity between major colors CvD [i_p, j], horizontal similarity between similar colors ChD [i_p, j], similarity between major different colors in the vertical direction CvN [i_p, j], horizontal similarity between different colors ChN [i_p, j] is calculated (corresponding to weighted addition of similarity components between the interpolation target pixel and the surrounding pixels as shown in FIG. 4).
[0177]
<< Peripheral addition method 1 >>
CvD [i_p, j] = (4 ・ CvD0 [i_p, j] + CvD0 [i_p-1, j-1] + CvD0 [i_p-1, j + 1]
+ CvD0 [i_p+ 1, j-1] + CvD0 [i_p+ 1, j + 1]) / 8 ・ ・ ・ Equation 93
ChD [i_p, j] = (4 ・ ChD0 [i_p, j] + ChD0 [i_p-1, j-1] + ChD0 [i_p-1, j + 1]
+ ChD0 [i_p+ 1, j-1] + ChD0 [i_p+ 1, j + 1]) / 8 ・ ・ ・ Equation 94
CvN [i_p, j] = (4 ・ CvN0 [i_p, j] + CvN0 [i_p-1, j-1] + CvN0 [i_p-1, j + 1]
+ CvN0 [i_p+ 1, j-1] + CvN0 [i_p+ 1, j + 1]) / 8 ・ ・ ・ Equation 95
ChN [i_p, j] = (4 ・ ChN0 [i_p, j] + ChN0 [i_p-1, j-1] + ChN0 [i_p-1, j + 1]
+ ChN0 [i_p+ 1, j-1] + ChN0 [i_p+ 1, j + 1]) / 8 ・ ・ ・ Equation 96
<< Peripheral addition method 2 >>
CvD [i_p, j] = (4 ・ CvD0 [i_p, j]
+2 ・ (CvD0 [i_p-1, j-1] + CvD0 [i_p+ 1, j-1] + CvD0 [i_p-1, j + 1] + CvD0 [i_p+ 1, j + 1])
+ CvD0 [i_p, j-2] + CvD0 [i_p, j + 2] + CvD0 [i_p-2, j] + CvD0 [i_p+ 2, j]) / 16 ・ ・ ・ Equation 97
ChD [i_p, j] = (4 ・ ChD0 [i_p, j]
+2 ・ (ChD0 [i_p-1, j-1] + ChD0 [i_p+ 1, j-1] + ChD0 [i_p-1, j + 1] + ChD0 [i_p+ 1, j + 1])
+ ChD0 [i_p, j-2] + ChD0 [i_p, j + 2] + ChD0 [i_p-2, j] + ChD0 [i_p+ 2, j]) / 16 ・ ・ ・ Formula 98
CvN [i_p, j] = (4 ・ CvN0 [i_p, j]
+2 ・ (CvN0 [i_p-1, j-1] + CvN0 [i_p+ 1, j-1] + CvN0 [i_p-1, j + 1] + CvN0 [i_p+ 1, j + 1])
+ CvN0 [i_p, j-2] + CvN0 [i_p, j + 2] + CvN0 [i_p-2, j] + CvN0 [i_p+ 2, j]) / 16 ・ ・ ・ Formula 99
ChN [i_p, j] = (4 ・ ChN0 [i_p, j]
+2 ・ (ChN0 [i_p-1, j-1] + ChN0 [i_p+ 1, j-1] + ChN0 [i_p-1, j + 1] + ChN0 [i_p+ 1, j + 1])
+ ChN0 [i_p, j-2] + ChN0 [i_p, j + 2] + ChN0 [i_p-2, j] + ChN0 [i_p+ 2, j]) / 16 ・ ・ ・ Equation 100
Next, the similarity between different colors of the fine part similarity in the vertical direction and the horizontal direction (hereinafter referred to as “fine part different color similarity”) and the same color similarity (hereinafter referred to as “fine part same color similarity”) Will be described.
[0178]
The interpolation processing unit 17 performs a plurality of types of fine part similarity components (Cvs1 [i] for the vertical direction and the horizontal direction defined by Expressions 29 to 34 of the first embodiment._p, j], Chs1 [i_p, j], Cvs2 [i_p, j], Chs2 [i_p, j], Cvs3 [i_p, j], Chs3 [i_p, j]).
Next, as shown in the following formulas 101 and 102, the interpolation processing unit 17 performs weighted addition of a plurality of types of similarity components between the same colors by weighting coefficients s1 and s2, and formulas 103 and 104. As shown in Fig. 1, the similarity component between different colors is expressed as CvsN0 [i_p, j] or ChsNO [i_p, j]. In addition, the vertical similarity component is also normalized by the distance between the sub-pixels by the following equations 101 and 103.
[0179]
CvsD0 [i_p, j] = (s1 ・ Cvs1 [i_p, j] / 4 + s2 ・ Cvs2 [i_p, j] / 4) / (s1 + s2) ... Equation 101
ChsD0 [i_p, j] = (s1 ・ Chs1 [i_p, j] + s2 ・ Chs2 [i_p, j]) / (s1 + s2) ... Formula 102
CvsN0 [i_p, j] = Cvs3 [i_p, j] / 2 ... Formula 103
ChsN0 [i_p, j] = Chs3 [i_p, j] ・ ・ ・ Formula 104
CvsD0 [i calculated in this way_p, j], ChsD0 [i_p, j], CvsN0 [i_p, j], ChsN0 [i_p, j] is the same color similarity CvsD [i_p, j], horizontal similarity between similar colors ChsD [i_p, j], the fineness similarity CvsN [i_p, j], the similarity between the fine parts in the horizontal direction ChsN [i_p, j], here, an example of calculating these similarities by peripheral addition is shown.
[0180]
In other words, the interpolation processing unit 17 obtains a value (CvsD0 [i_p, j], CvsD0 [i_p-1, j-1], CvsD0 [i_p-1, j + 1], CvsD0 [i_p+ 1, j-1], CvsD0 [i_p+ 1, j + 1], etc.) in consideration of the continuity of similarity and weighted addition as in the following << peripheral addition method 1 >> or << peripheral addition method 2 >> Similarity between same colors in fine parts CvsD [i_p, j], horizontal similarity between similar colors ChsD [i_p, j], the fineness similarity CvsN [i_p, j], the similarity between the fine parts in the horizontal direction ChsN [i_p, j] is calculated (corresponding to weighted addition of similarity components between the interpolation target pixel and the surrounding pixels as shown in FIG. 4).
[0181]
<< Peripheral addition method 1 >>
CvsD [i_p, j] = (4 ・ CvsD0 [i_p, j] + CvsD0 [i_p-1, j-1] + CvsD0 [i_p-1, j + 1]
+ CvsD0 [i_p+ 1, j-1] + CvsD0 [i_p+ 1, j + 1]) / 8 ・ ・ ・ Formula 105
ChsD [i_p, j] = (4 ・ ChsD0 [i_p, j] + ChsD0 [i_p-1, j-1] + ChsD0 [i_p-1, j + 1]
+ ChsD0 [i_p+ 1, j-1] + ChsD0 [i_p+ 1, j + 1]) / 8 ・ ・ ・ Formula 106
CvsN [i_p, j] = (4 ・ CvsN0 [i_p, j] + CvsN0 [i_p-1, j-1] + CvsN0 [i_p-1, j + 1]
+ CvsN0 [i_p+ 1, j-1] + CvsN0 [i_p+ 1, j + 1]) / 8 ・ ・ ・ Formula 107
ChsN [i_p, j] = (4 ・ ChsN0 [i_p, j] + ChsN0 [i_p-1, j-1] + ChsN0 [i_p-1, j + 1]
+ ChsN0 [i_p+ 1, j-1] + ChsN0 [i_p+ 1, j + 1]) / 8 ・ ・ ・ Formula 108
<< Peripheral addition method 2 >>
CvsD [i_p, j] = (4 ・ CvsD0 [i_p, j] +2 ・ (CvsD0 [i_p-1, j-1] + CvsD0 [i_p+ 1, j-1]
+ CvsD0 [i_p-1, j + 1] + CvsD0 [i_p+ 1, j + 1]) + CvsD0 [i_p, j-2] + CvsD0 [i_p, j + 2]
+ CvsD0 [i_p-2, j] + CvsD0 [i_p+ 2, j]) / 16 ・ ・ ・ Formula 109
ChsD [i_p, j] = (4 ・ ChsD0 [i_p, j] +2 ・ (ChsD0 [i_p-1, j-1] + ChsD0 [i_p+ 1, j-1]
+ ChsD0 [i_p-1, j + 1] + ChsD0 [i_p+ 1, j + 1]) + ChsD0 [i_p, j-2] + ChsD0 [i_p, j + 2]
+ ChsD0 [i_p-2, j] + ChsD0 [i_p+ 2, j]) / 16 ・ ・ ・ Formula 110
CvsN [i_p, j] = (4 ・ CvsN0 [i_p, j] +2 ・ (CvsN0 [i_p-1, j-1] + CvsN0 [i_p+ 1, j-1]
+ CvsN0 [i_p-1, j + 1] + CvsN0 [i_p+ 1, j + 1]) + CvsN0 [i_p, j-2] + CvsN0 [i_p, j + 2]
+ CvsN0 [i_p-2, j] + CvsN0 [i_p+ 2, j]) / 16 ・ ・ ・ Formula 111
ChsN [i_p, j] = (4 ・ ChsN0 [i_p, j] +2 ・ (ChsN0 [i_p-1, j-1] + ChsN0 [i_p+ 1, j-1]
+ ChsN0 [i_p-1, j + 1] + ChsN0 [i_p+ 1, j + 1]) + ChsN0 [i_p, j-2] + ChsN0 [i_p, j + 2]
+ ChsN0 [i_p-2, j] + ChsN0 [i_p+ 2, j]) / 16 ・ ・ ・ Formula 112
By the way, as described above, the similarity between major different colors and the similarity between different fine parts are calculated by comparing the color information of adjacent pixels in the vertical direction or the horizontal direction. The degree of similarity can be determined at a distance interval shorter than the degree or the similarity between the fine parts. That is, the similarity between different colors reflects a finer image structure than the similarity between the same colors.
[0182]
In particular, since the similarity between different colors is calculated on the assumption that all the color information of different color components represents the same luminance information, the determination of the strength of similarity using the similarity between different colors is achromatic. In part, the reliability is high. On the other hand, the determination of the strength of similarity using the similarity between the same colors is generally reliable for both the chromatic and achromatic parts, but when the similarity between the different colors is used in the detailed part of the image Compared to reliability.
[0183]
Therefore, in order to perform highly reliable similarity determination for the entire image to be interpolated, the entire image is divided into an achromatic portion and a chromatic portion, and a similarity suitable for each portion is set. The method used is excellent. In the G interpolation process in the third embodiment, in order to realize such a highly reliable similarity determination, an index HV [i is determined depending on whether or not an image in the vicinity of the interpolation target pixel is an achromatic color part._p, j] is switched.
[0184]
Hereinafter, although the G interpolation process in the third embodiment will be described, detailed description of the same operations as those in the first embodiment will be omitted.
First, the interpolation processing unit 17 calculates a color index indicating the presence or absence of a color in the local region to which the interpolation target pixel belongs (S1 in FIG. 19), and based on the color index, whether the image near the interpolation target pixel is an achromatic color part. It is determined whether or not (S2 in FIG. 19).
[0185]
Note that local color difference information can be used as the color index. However, the similarity between the main different colors and the similarity between the fine parts which are calculated as described above are determined at the same time as the strength of the similarity. Since the color difference information is reflected, it is possible to directly use the main different color similarity and the fine portion different color similarity as the color index.
In the third embodiment, as the color index, either the main different color similarity or the fine part different color similarity in the vertical direction and the horizontal direction can be used, but here, the main different color similarity is used. An example of the case is shown.
[0186]
However, the similarity between major different colors indicates that the smaller the value is, the stronger the similarity is. Therefore, when the similarity between major different colors in the vertical and horizontal directions is a large value, both the vertical and horizontal directions in the achromatic part This means that the similarity to is weak, or the image in the vicinity of the interpolation target pixel is a chromatic part. Conversely, if the similarity between major different colors in at least one of the vertical and horizontal directions is a relatively small value, the image in the vicinity of the interpolation target pixel is an achromatic part and there is a direction with strong similarity Means that
[0187]
That is, the interpolation processing unit 17 performs similarity CvN [i between main different colors in the vertical direction._p, j] and the horizontal similarity between the different colors ChN [i_p, j] and determine whether or not the following condition 1 is satisfied for the threshold values ThNv and ThNh, and if condition 1 is satisfied, determine that the image in the vicinity of the interpolation target pixel is an achromatic portion, When the condition 1 is not satisfied, it is determined that the image in the vicinity of the interpolation target pixel is a chromatic part.
[0188]
CvN [i_p, j] ≦ ThNv or ChN [i_p, j] ≦ ThNh ・ ・ ・ Condition 1
However, the threshold values ThNv and ThNh assume values of about 10 or less when the number of gradations is 256.
Then, when the image in the vicinity of the pixel to be interpolated is an achromatic color part, the interpolation processing unit 17 refers to an index HV [i using a similarity between different colors described later._p, j] setting process (S3 in FIG. 19), and if the image near the interpolation target pixel is a chromatic part, an index HV [i using the similarity between same colors described later will be described._p, j] setting process >> (S4 in FIG. 19).
[0189]
By such processing, the index HV [i_p, j] is set, the interpolation processing unit 17 performs the index HV [i as in the first embodiment (corresponding to S10 to S13 in FIG. 2)._p, j], the green interpolation values G [i, j] and G [i + 1, j] of the two subpixels are calculated (S5 to S8 in FIG. 19).
Here, << index HV [i using the similarity between different colors_p, j] setting processing and << index HV [i using similarity between same colors_p, j] setting process >>.
[0190]
《Indicator HV [i_p, j] setting process >>
First, the interpolation processing unit 17 performs similarity CvN [i between main different colors in the vertical direction._p, j] and the horizontal similarity between the different colors ChN [i_p, j] (FIG. 20S1), and it is determined whether or not the direction based on these similarities (hereinafter referred to as “similarity between main different colors”) is unknown (FIG. 20S2). ).
[0191]
For example, such a determination can be realized by determining whether or not the following condition 2 is satisfied for the threshold Th0.
| CvN [i_p, j] -ChN [i_p, j] | ≦ Th0 ・ ・ ・ Condition 2
Then, when the direction in which the similarity between the main different colors is strong is unknown as the determination result, the interpolation processing unit 17 determines the similarity between the fine portion different colors CvsN [i in the vertical direction._p, j] and the similarity between the fine parts in the horizontal direction ChsN [i_p, j] (FIG. 20S3), and determines whether or not the direction based on these similarities (hereinafter referred to as “similarity between fine parts different colors”) is unknown (FIG. 20). 20S4).
[0192]
For example, such a determination can be realized by determining whether or not the following condition 3 is satisfied for the threshold Ths0.
| CvsN [i_p, j] -ChsN [i_p, j] | ≦ Ths0 ・ ・ ・ Condition 3
And as a result of such determination, the interpolation processing unit 17 corresponds to the region 1 (1) in FIG. 22 when the direction in which the similarity between the fine portions is different is unknown (when the condition 2 and the condition 3 are satisfied). ), The image near the pixel to be interpolated is an achromatic color part, and it is determined that the similarity cannot be distinguished between the vertical and horizontal directions, and the index HV [i_p, j] is set to 0 (S5 in FIG. 20). On the other hand, if the direction in which the fine part different color similarity is strong can be determined, it is determined whether the vertical fine part different color similarity is strong (FIG. 20 S6).
[0193]
For example, such a determination can be realized by determining whether or not the following condition 4 is satisfied.
CvsN [i_p, j] <ChsN [i_p, j] ... Condition 4
Then, as a result of such determination, the interpolation processing unit 17 determines that the similarity between the fine parts in the vertical direction is strong (conditions 2 and 4 are satisfied, and condition 3 is not satisfied: region 2 (1 )), The image in the vicinity of the pixel to be interpolated is an achromatic portion, and it is determined that the similarity in the vertical direction is strong, and the index HV [i_p, j] is set to 1 (S7 in FIG. 20). On the other hand, when the similarity between different colors in the vertical direction is not strong (when condition 2 is satisfied and conditions 3 and 4 are not satisfied: corresponding to region 1 (2) in FIG. 22), the index HV [i_p, j] is set to 0 (S8 in FIG. 20).
[0194]
In addition, when the direction in which the similarity between main different colors is strong can be determined as the determination result in FIG. 20S2, the interpolation processing unit 17 determines whether the similarity between main different colors in the vertical direction is strong (S9 in FIG. 20).
For example, such a determination can be realized by determining whether or not the following condition 5 is satisfied.
[0195]
CvN [i_p, j] <ChN [i_p, j] ... Condition 5
Then, as a result of such determination, the interpolation processing unit 17 determines that the similarity between major different colors in the vertical direction is strong (when condition 2 is not satisfied and condition 5 is satisfied: corresponding to region 2 (2) in FIG. 22). , Indicator HV [i_p, j] is set to 1 (S10 in FIG. 20). On the other hand, when the similarity between main different colors in the vertical direction is not strong (when conditions 2 and 5 are not satisfied: corresponding to region 3 in FIG. 22), the image near the interpolation target pixel is an achromatic portion, Judging that the direction is very similar, the indicator HV [i_p, j] is set to -1 (S11 in FIG. 20).
[0196]
《Indicator HV [i_p, j] setting process >>
First, the interpolation processing unit 17 calculates the similarity CvD [i between main same colors in the vertical direction._p, j] and the horizontal similar similarity ChD [i_p, j] is calculated (FIG. 21S1), and it is determined whether the direction based on these similarities (hereinafter referred to as “similar similarity between main colors”) is unknown (FIG. 21S2). ).
[0197]
For example, such a determination can be realized by determining whether or not the following condition 6 is satisfied for the threshold Th1.
| CvD [i_p, j] -ChD [i_p, j] | ≦ Th1 ・ ・ ・ Condition 6
Then, as a result of such determination, if the direction in which the similarity between the main same colors is strong is unknown, the interpolation processing unit 17 determines the similarity between the fine portions in the vertical direction CvsD [i_p, j] and the horizontal similarity between similar parts ChsD [i_p, j] is calculated (S3 in FIG. 21), and it is determined whether the direction based on these similarities (hereinafter referred to as “similar similarity between fine parts”) is unknown (FIG. 21). 21S4).
[0198]
For example, such a determination can be realized by determining whether or not the following condition 7 is satisfied for the threshold Ths1.
| CvsD [i_p, j] -ChsD [i_p, j] | ≦ Ths1 ・ ・ ・ Condition 7
And as a result of such determination, the interpolation processing unit 17 corresponds to the region 4 (1) in FIG. 22 when the direction in which the similarity between the fine portions is the same is unknown (when the conditions 6 and 7 are satisfied). ), The image near the pixel to be interpolated is the chromatic part, and it is determined that the similarity cannot be distinguished between the vertical and horizontal directions, and the index HV [i_p, j] is set to 0 (S5 in FIG. 21). On the other hand, when the direction where the similarity between the fine parts is the same can be determined, it is determined whether the similarity between the fine parts is the same in the vertical direction (S6 in FIG. 21).
[0199]
For example, such a determination can be realized by determining whether or not the following condition 8 is satisfied.
CvsD [i_p, j] <ChsD [i_p, j] ... Condition 8
Then, as a result of such determination, the interpolation processing unit 17 determines that the similarity between the fine parts in the vertical direction is strong (conditions 6 and 8 are satisfied, and condition 7 is not satisfied: region 5 (1 in FIG. 22). )), The image in the vicinity of the interpolation target pixel is a colored portion, and it is determined that the similarity in the vertical direction is strong, and the index HV [i_p, j] is set to 1 (S7 in FIG. 21). On the other hand, if the similarity between the same colors in the fine parts in the vertical direction is not strong (when condition 6 is satisfied and conditions 7 and 8 are not satisfied: corresponding to region 4 (2) in FIG. 22), index HV [i_p, j] is set to 0 (S8 in FIG. 21).
[0200]
Further, when the direction in which the similarity between the main same colors is strong can be determined as the determination result in FIG. 21S2, the interpolation processing unit 17 determines whether the similarity in the main same color in the vertical direction is strong (FIG. 21 S9).
For example, such a determination can be realized by determining whether or not the following condition 9 is satisfied.
[0201]
CvD [i_p, j] <ChD [i_p, j] ... Condition 9
Then, as a result of such determination, the interpolation processing unit 17 determines that the similarity between main colors in the vertical direction is strong (when condition 6 is not satisfied and condition 9 is satisfied: corresponding to region 5 (2) in FIG. 22). , Indicator HV [i_p, j] is set to 1 (S10 in FIG. 21). On the other hand, when the similarity between the main colors in the vertical direction is not strong (when conditions 6 and 9 do not hold: corresponding to region 6 in FIG. 22), the image in the vicinity of the interpolation target pixel is a chromatic part, and the horizontal direction And the index HV [i_p, j] is set to -1 (FIG. 21 S11).
[0202]
By the way, << index HV [i using the similarity between different colors_p, j] setting processing and << index HV [i using similarity between same colors_p, j] setting process >>, the threshold values Th0, Ths0, Th1, Ths1 have strong similarities due to noise when the difference between the vertical similarity and the horizontal similarity is small. It plays a role to avoid being mistaken. Therefore, for a noisy color image, the accuracy of similarity determination can be increased by setting the threshold values Th0, Ths0, Th1, and Ths1 high.
[0203]
As described above, in the third embodiment, the entire image is divided into the achromatic portion and the chromatic portion by using the similarity between different colors reflecting the local color difference information, and the similarity suitable for each portion is used. Since the degree of similarity can be determined based on the degree, the interpolation amount can be calculated with higher accuracy than in the conventional technique.
In the third embodiment, when determining the strength of similarity, the similarity between different colors and the similarity between the same colors are switched. As the similarity used for determining the similarity, the similarity between different colors is used. In addition to completely switching between the same-color similarity, the achromatic part increases the similar-color similarity addition ratio, and the chromatic part increases the same-color similarity addition ratio. Similarity obtained by weighted addition of.
[0204]
In the third embodiment, the color difference included in the similarity between different colors is used as a method for checking the presence / absence of a local color, but other color indexes such as a color ratio may be used.
By the way, the index HV [i set in the third embodiment_p, j] can be used in the second embodiment when classifying the strength of similarity of pixels to be interpolated (corresponding to S15 in FIG. 13). That is, instead of the processes of FIGS. 12S1 to S4 and S11 to S14 of the second embodiment, the processes of FIGS. 19S1 to S4, FIG. 20 and FIG. 21 of the third embodiment can be performed.
[0205]
<< Fourth Embodiment >>
Hereinafter, the operation of the fourth embodiment will be described, but a rough flow of the G interpolation processing in the fourth embodiment (a flow of calculating a similarity and setting an index and calculating an interpolation value based on the index) Since this is the same as in the first embodiment, the illustration is omitted. Further, the RB interpolation process in the fourth embodiment can be performed in the same manner as in the first embodiment, and thus the description thereof is omitted.
[0206]
Note that the difference in the G interpolation processing between the fourth embodiment and the first embodiment is that the first embodiment determines the similarity in the vertical and horizontal directions for each interpolation target pixel and the index HV [i_p, j] is set, the fourth embodiment determines the similarity in the vertical and horizontal directions and the indices HV [i, j], HV [i + 1, j] is set.
That is, in the fourth embodiment, the main similarity Cv [i in the vertical direction in the interpolation target pixel calculated in the first embodiment._p, j], the main similarity Cv [i, j] and Cv [i + 1, j] in the vertical direction in the two interpolation target sub-pixels are calculated, and the main similarity in the horizontal direction is calculated. The same applies to the fine part similarity in the vertical direction and the fine part similarity in the horizontal direction.
[0207]
Therefore, here, as the operation of the fourth embodiment, first, the main similarity (Cv [i, j], Ch [i, j], Cv [i + 1, j], Ch in the vertical direction and the horizontal direction). [i + 1, j]) and fine part similarity (Cvs [i, j], Chs [i, j], Cvs [i + 1, j], Chs [i + 1, j]) Processing will be described.
However, in the arithmetic expression described later, the arithmetic expression for the sub-pixel located at the coordinate [i + 1, j] in FIG. 28 (2) is changed to i in the arithmetic expression for the sub-pixel located at the coordinate [i, j]. When it can be realized by substituting +1, only the arithmetic expression for the subpixel located at the coordinates [i, j] is described, and the arithmetic expression for the subpixel located at the coordinates [i + 1, j] is omitted. To do.
[0208]
First, the process of calculating the main similarity Cv [i, j] in the vertical direction and the main similarity Ch [i, j] in the horizontal direction for the sub-pixel located at the coordinates [i, j] will be described (coordinates [ The main similarity Cv [i + 1, j] in the vertical direction and the main similarity Ch [i + 1, j] in the horizontal direction for the subpixel located at i + 1, j] are expressed by the following Expressions 113 to 113. It can be calculated by substituting i + 1 for i of 131).
[0209]
The interpolation processing unit 17 calculates a plurality of types of main similarity components for the vertical direction and the horizontal direction defined by the following Expressions 113 to 124. In the following Expressions 113 to 124, the main similarity component is configured with a positional relationship that is symmetrical between the vertical direction and the horizontal direction with respect to the distance.
Main similarity component between GG in the vertical direction:
Cv1 [i, j] = | G [i, j-1] -G [i, j + 1] |
Horizontal similarity component between GG:
Ch1 [i, j] = | G [i-2, j] -G [i + 2, j] |
Main similarity component between BB (RR) in the vertical direction:
Cv2 [i, j] = (| Z [i-2, j-1] -Z [i-2, j + 1] | + | Z [i + 2, j-1] -Z [i + 2, j + 1] |) / 2 ... Formula 115
Main similarity component between horizontal BB (RR):
Ch2 [i, j] = (| Z [i-2, j-1] -Z [i + 2, j-1] | + | Z [i-2, j + 1] -Z [i + 2, j + 1] |) / 2 ・ ・ ・ Formula 116
Main similarity component between vertical RR (BB):
Cv3 [i, j] = (| Z [i, j-2] -Z [i, j] | + | Z [i, j + 2] -Z [i, j] |) / 2 117
Main similarity component between horizontal RR (BB):
Ch3 [i, j] = (| Z [i-4, j] -Z [i, j] | + | Z [i + 4, j] -Z [i, j] |) / 2 118
Major similarity component between longitudinal GR (GB):
Cv4 [i, j] = (| G [i, j-1] -Z [i, j] | + | G [i, j + 1] -Z [i, j] |) / 2 119
Main similarity component between horizontal GR (GB):
Ch4 [i, j] = (| G [i-2, j] -Z [i, j] | + | G [i + 2, j] -Z [i, j] |) / 2 120
Main similarity component between vertical BG (RG):
Cv5 [i, j] = (| Z [i-2, j-1] -G [i-2, j] | + | Z [i-2, j + 1] -G [i-2, j] |
+ | Z [i + 2, j-1] -G [i + 2, j] | + | Z [i + 2, j + 1] -G [i + 2, j] |) / 4 Formula 121
Main similarity component between lateral BG (RG):
Ch5 [i, j] = (| Z [i-2, j-1] -G [i, j-1] | + | Z [i-2, j + 1] -G [i, j + 1] |
+ | Z [i + 2, j-1] -G [i, j-1] | + | Z [i + 2, j + 1] -G [i, j + 1] |) / 4 Formula 122
Main similarity component between luminance in vertical direction:
Cv6 [i, j] = (| Y [i, j-1] -Y [i, j] | + | Y [i, j + 1] -Y [i, j] |) / 2 one two Three
Main similarity component between luminance in horizontal direction:
Ch6 [i, j] = (| Y [i-2, j] -Y [i, j] | + | Y [i + 2, j] -Y [i, j] |) / 2 124
However, in Expression 123 and Expression 124, Y [i, j] is a value calculated by Expression 125 below, and the color information of the color component of the sub-pixel located in the vicinity of the interpolation target sub-pixel is R: G This corresponds to the luminance generated by the filtering process that averages at a ratio of: B = 1: 2: 1.
[0210]
Y [i, j] = (4 ・ (A [i, j] + A [i-1, j] + A [i + 1, j])
+2 ・ (A [i, j-1] + A [i-1, j-1] + A [i + 1, j-1]
+ A [i, j + 1] + A [i-1, j + 1] + A [i + 1, j + 1]
+ A [i-2, j] + A [i + 2, j])
+ A [i-2, j-1] + A [i-2, j + 1] + A [i + 2, j-1] + A [i + 2, j + 1]) / 32 ・ ・ ・Formula 125
However, in Expression 125, A [i, j] represents arbitrary color information on the sub-pixel type Bayer array, and takes a value of G or Z depending on the arrangement location. FIG. 23 is a diagram illustrating a filter for generating luminance.
[0211]
In addition, although the element of the similarity which comprises each main similarity component mentioned above is calculated using the absolute value of a difference, you may calculate by the square of an absolute value, a power, etc. Next, as shown in the following Expression 126 and Expression 127, the interpolation processing unit 17 uses a plurality of types of weighting coefficients a1, a2, a3, a4, a5, and a6 that play the same role as in the first embodiment. Are weighted for each direction.
[0212]
Cv0 [i, j] = (a1 ・ Cv1 [i, j] + a2 ・ Cv2 [i, j] + a3 ・ Cv3 [i, j] + a4 ・ Cv4 [i, j]
+ a5 ・ Cv5 [i, j] + a6 ・ Cv6 [i, j]) / (a1 + a2 + a3 + a4 + a5 + a6)
Ch0 [i, j] = (a1 ・ Ch1 [i, j] + a2 ・ Ch2 [i, j] + a3 ・ Ch3 [i, j] + a4 ・ Ch4 [i, j]
+ a5 ・ Ch5 [i, j] + a6 ・ Ch6 [i, j]) / (a1 + a2 + a3 + a4 + a5 + a6)
Cv0 [i, j] and Ch0 [i, j] calculated in this way are directly used as the main similarity Cv [i, j] in the vertical direction and the main similarity Ch [ i, j]. Here, an example is shown in which these similarities are calculated by peripheral addition.
[0213]
That is, the interpolation processing unit 17 performs weighted addition for each direction of main similarity components in the interpolation target subpixel and a subpixel located in the vicinity of the interpolation target subpixel (hereinafter referred to as “peripheral subpixel”). The resulting values (Cv0 [i, j], Cv0 [i-1, j-1], Cv0 [i-1, j + 1], Cv0 [i + 1, j-1], Cv0 [i + 1, j + 1] etc.) is weighted and added as in the following << peripheral addition method 1 >> or << peripheral addition method 2 >>, and the vertical main similarity Cv [i, j] and horizontal The main similarity Ch [i, j] in the direction is calculated.
[0214]
<< Peripheral addition method 1 >>
Cv [i, j] = (4 ・ Cv0 [i, j] + Cv0 [i-2, j-1] + Cv0 [i-2, j + 1] + Cv0 [i + 2, j-1]
+ Cv0 [i + 2, j + 1]) / 8 ・ ・ ・ Equation 128
Ch [i, j] = (4 ・ Ch0 [i, j] + Ch0 [i-2, j-1] + Ch0 [i-2, j + 1] + Ch0 [i + 2, j-1]
+ Ch0 [i + 2, j + 1]) / 8 ・ ・ ・ Equation 129
<< Peripheral addition method 2 >>
Cv [i, j] = (4 ・ Cv0 [i, j]
+2 ・ (Cv0 [i-2, j-1] + Cv0 [i + 2, j-1] + Cv0 [i-2, j + 1] + Cv0 [i + 2, j + 1])
+ Cv0 [i, j-2] + Cv0 [i, j + 2] + Cv0 [i-4, j] + Cv0 [i + 4, j]) / 16 Equation 130
Ch [i, j] = (4 ・ Ch0 [i, j]
+2 ・ (Ch0 [i-2, j-1] + Ch0 [i + 2, j-1] + Ch0 [i-2, j + 1] + Ch0 [i + 2, j + 1])
+ Ch0 [i, j-2] + Ch0 [i, j + 2] + Ch0 [i-4, j] + Ch0 [i + 4, j]) / 16 ... Equation 131
Note that << Peripheral addition method 1 >> corresponds to performing weighted addition of main similarity components in the interpolation target subpixel and the peripheral subpixel as shown in FIG. 24 (1). << Peripheral addition method 2 >> Corresponds to performing weighted addition of the main similarity components in the interpolation target subpixel and the peripheral subpixel as shown in FIG.
[0215]
Next, the vertical fine portion similarity Cvs [i, j] and the horizontal fine portion similarity Chs [i, j] for the subpixel located at the coordinates [i, j], and the coordinates [i + 1, Processing for calculating the vertical fine portion similarity Cvs [i + 1, j] and the horizontal fine portion similarity Chs [i + 1, j] for the subpixel located at j] will be described.
[0216]
The interpolation processing unit 17 calculates a plurality of types of fine part similarity components for the vertical direction and the horizontal direction defined by the following Expressions 132 to 143. In the following Expressions 132 to 143, the fine part similarity component is configured in an asymmetric positional relationship between the vertical direction and the horizontal direction with respect to the distance.
Fine part similarity component between RR (BB) in the vertical direction:
Cvs1 [i, j] = (| Z [i, j-2] -Z [i, j] | + | Z [i, j + 2] -Z [i, j] |) / 2 132
Cvs1 [i + 1, j] = (| Z [i + 1, j-2] -Z [i + 1, j] | + | Z [i + 1, j + 2] -Z [i + 1, j] |) / 2 ・ ・ ・ Formula 133
Fine part similarity component between horizontal RR (BB):
Chs1 [i, j] = | Z [i, j] -Z [i + 1, j] |
Chs1 [i + 1, j] = | Z [i, j] -Z [i + 1, j] |
Fine part similarity component between GG in the vertical direction:
Cvs2 [i, j] = | G [i, j-1] -G [i, j + 1] |
Cvs2 [i + 1, j] = | G [i + 1, j-1] -G [i + 1, j + 1] |
Lateral GG fine part similarity component:
Chs2 [i, j] = (| G [i, j-1] -G [i + 1, j-1] | + | G [i, j + 1] -G [i + 1, j + 1] |) / 2 ・ ・ ・ Formula 138
Chs2 [i + 1, j] = (| G [i, j-1] -G [i + 1, j-1] | + | G [i, j + 1] -G [i + 1, j + 1] |) / 2 ・ ・ ・ Equation 139
Fine part similarity component between vertical GR (GB):
Cvs3 [i, j] = (| G [i, j-1] -Z [i, j] | + | G [i, j + 1] -Z [i, j] |) / 2 140
Cvs3 [i + 1, j] = (| G [i + 1, j-1] -Z [i + 1, j] | + | G [i + 1, j + 1] -Z [i + 1, j] |) / 2 ・ ・ ・ Formula 141
Fine part similarity component between horizontal GR (GB):
Chs3 [i, j] = | G [i-1, j] -Z [i, j] |
Chs3 [i + 1, j] = | G [i + 2, j] -Z [i + 1, j] |
In addition, although the element of the similarity which comprises each fine part similarity component mentioned above is calculated using the absolute value of a difference, you may calculate by the square of an absolute value, a power, etc.
[0217]
Next, as shown in the following formulas 144 and 145, the interpolation processing unit 17 weight-adds a plurality of types of fine part similarity components for each direction using the weighting coefficients s1, s2, and s3 (coordinates [i + Similarly, a plurality of types of fine part similarity components for sub-pixels located at [1, j] can be weighted in different directions. In Formula 144, the vertical similarity component is normalized by the distance between subpixels.
[0218]
Cvs0 [i, j] = (s1, Cvs1 [i, j] / 4 + s2, Cvs2 [i, j] / 4 + s3, Cvs3 [i, j] / 2)
/ (s1 + s2 + s3) ... Formula 144
Chs0 [i, j] = (s1 ・ Chs1 [i, j] + s2 ・ Chs2 [i, j] + s3 ・ Chs3 [i, j])
/ (s1 + s2 + s3) ... Formula 145
Cvs0 [i, j] and Chs0 [i, j] calculated in this way are used as they are for the fine portion similarity Cvs [i, j] in the vertical direction and the fine portion similarity in the horizontal direction of the sub-pixel to be interpolated. Chs [i, j] can be used, but here, an example is shown in which these similarities are calculated by peripheral addition.
[0219]
That is, the interpolation processing unit 17 obtains values (Cvs0 [i, j], Cvs0 [i−1, j−1) obtained by weighted addition for each direction of the fine portion similarity component in the interpolation target subpixel and the peripheral subpixel. ], Cvs0 [i-1, j + 1], Cvs0 [i + 1, j-1], Cvs0 [i + 1, j + 1], etc.) As in Method 2, the weighted addition is performed to calculate the vertical fine portion similarity Cvs [i, j] and the horizontal fine portion similarity Chs [i, j] (coordinates [i + 1, j] Similarly, the fine part similarity with respect to the sub-pixel located at is also calculated).
[0220]
<< Peripheral addition method 1 >>
Cvs [i, j] = (4 ・ Cvs0 [i, j] + Cvs0 [i-2, j-1] + Cvs0 [i-2, j + 1] + Cvs0 [i + 2, j-1]
+ Cvs0 [i + 2, j + 1]) / 8 ・ ・ ・ Equation 146
Chs [i, j] = (4 ・ Chs0 [i, j] + Chs0 [i-2, j-1] + Chs0 [i-2, j + 1] + Chs0 [i + 2, j-1]
+ Chs0 [i + 2, j + 1]) / 8 ・ ・ ・ Formula 147
<< Peripheral addition method 2 >>
Cvs [i, j] = (4 ・ Cvs0 [i, j]
+2 ・ (Cvs0 [i-2, j-1] + Cvs0 [i + 2, j-1] + Cvs0 [i-2, j + 1] + Cvs0 [i + 2, j + 1])
+ Cvs0 [i, j-2] + Cvs0 [i, j + 2] + Cvs0 [i-4, j] + Cvs0 [i + 4, j]) / 16 Equation 148
Chs [i, j] = (4 ・ Chs0 [i, j]
+2 ・ (Chs0 [i-2, j-1] + Chs0 [i + 2, j-1] + Chs0 [i-2, j + 1] + Chs0 [i + 2, j + 1])
+ Chs0 [i, j-2] + Chs0 [i, j + 2] + Chs0 [i-4, j] + Chs0 [i + 4, j]) / 16 Equation 149
By the way, the determination of the main similarity and the fine part similarity and the setting of the indices HV [i, j] and HV [i + 1, j] in the two interpolation target subpixels of the fourth embodiment are as described above. By using the calculated main similarity and fine part similarity, the calculation can be performed in the same manner as the processing in the interpolation target pixel of the first embodiment. That is, in the description of FIGS. 2S2 to S6, S8, and S9 of the first embodiment, the coordinates [i of FIG._p, j], instead of the interpolation target pixel located at the coordinates [i, j] and [i + 1, j] in FIG. 28 (2).
[0221]
When the setting of the indices HV [i, j] and HV [i + 1, j] is completed in this way, the interpolation processing unit 17 performs green interpolation values of the two interpolation target subpixels as follows. G [i, j] and G [i + 1, j] are calculated.
When HV [i, j] = 1 (similar in the vertical direction) G [i, j] = GvL
When HV [i + 1, j] = 1 (similar in the vertical direction) G [i + 1, j] = GvR
When HV [i, j] = 0 (similar in strength in the vertical and horizontal directions, or weak)
G [i, j] = (GvL + GhL) / 2
When HV [i + 1, j] = 0 (similar or weak in the vertical and horizontal directions)
G [i + 1, j] = (GvR + GhR) / 2
When HV [i, j] =-1 (similar in horizontal direction) G [i, j] = GhL
When HV [i + 1, j] =-1 (Strong similarity in the horizontal direction) G [i + 1, j] = GhR
However, GvL, GvR, GhL, and GhR are values calculated as in “average interpolation” or “color difference interpolation” in the first embodiment.
[0222]
As described above, in the fourth embodiment, the similarity can be calculated for each interpolation target sub-pixel, and the interpolation value can be calculated according to the similarity determination result based on the similarity.
[0223]
In the fourth embodiment, as in the first embodiment, the main similarity and the fine part similarity are calculated in the vertical direction and the horizontal direction, and the green interpolation value is calculated. For example, as shown in Expression 150 to Expression 153, the main similarity C45 [i, j] and C45 [i + 1, j] in the oblique 45 degree direction and the main similarity C135 [i, j in the oblique 135 degree direction are shown. ], C135 [i + 1, j] is calculated, and based on these similarities, as in the second embodiment, the diagonal main similarity of the interpolation target subpixel is determined to calculate the green interpolation value. May be.
[0224]
C45 [i, j] = (| R [i + 4, j-2] -R [i, j] |-| R [i-4, j + 2] -R [i, j] |) / 2 ... Formula 150
C45 [i + 1, j] = (| R [i + 5, j-2] -R [i + 1, j] |-| R [i-3, j + 2] -R [i + 1, j] |) / 2 ・ ・ ・ Formula 151
C135 [i, j] = (| R [i-4, j-2] -R [i, j] |-| R [i + 4, j + 2] -R [i, j] |) / 2 ... Formula 152
C135 [i + 1, j] = (| R [i-3, j-2] -R [i + 1, j] |-| R [i + 5, j + 2] -R [i + 1, j] |) / 2 ・ ・ ・ Formula 153
In the fourth embodiment, the values of the indices HV [i, j] and HV [i + 1, j] are set as in the first embodiment, but the index HV [i, j ] And HV [i + 1, j] may be calculated in the same manner as in the third embodiment. That is, as in the third embodiment, the main similarity is divided into the main different color similarity and the main same color similarity, and the fine part similarity is divided into the fine part different color similarity and the fine part same color similarity. The process of switching and using the similarity between different colors and the similarity between the same colors is represented by coordinates [i in FIG._p, j], instead of the interpolation target pixel located at the coordinates [i, j] and [i + 1, j] in FIG. Just do it.
[0225]
<< Fifth Embodiment >>
The operation of the fifth embodiment will be described below. Note that the RB interpolation processing in the fifth embodiment can be performed in the same manner as in the first embodiment, and thus description thereof is omitted here.
FIG. 25 is an operation flowchart of the interpolation processing unit 17 in the fifth embodiment, and shows the operation of the interpolation processing unit 17 in the G interpolation processing. However, FIG. 25 shows an example in which a sub-pixel located on the left side of a pixel on the Bayer array is an interpolation target.
[0226]
The G interpolation process in the fifth embodiment will be described below based on FIG.
First, the interpolation processing unit 17 calculates the vertical and horizontal similarities of the interpolation target sub-pixel (S1 in FIG. 25).
Here, details of the process of calculating the vertical similarity Cv [i, j] and the horizontal similarity Ch [i, j] in the interpolation target subpixel located at the coordinates [i, j] will be described.
[0227]
The interpolation processing unit 17 calculates a plurality of types of similarity components for the vertical direction and the horizontal direction defined by Expressions 154 to 166. In Expressions 154 to 166, the similarity component between neighboring pixels is calculated regardless of the symmetrical relationship as in the above-described embodiment.
Vertical GG similarity component:
Cv1 [i, j] = | G [i, j-1] -G [i, j + 1] |
Horizontal GG similarity component:
Gh1 [i, j] = | G [i-1, j] -G [i + 2, j] |
Vertical BB (RR) similarity component:
Cv2 [i, j] = (| Z [i-1, j-1] -Z [i-1, j + 1] | + | Z [i + 2, j-1] -Z [i + 2, j + 1] |) / 2 ・ ・ ・ Formula 156
Horizontal BB (RR) similarity component:
Ch2 [i, j] = (| Z [i-1, j-1] -Z [i + 2, j-1] | + | Z [i-1, j + 1] -Z [i + 2, j + 1] |) / 2 ... Formula 157
Vertical RR (BB) similarity component:
Cv3 [i, j] = (| Z [i, j-2] -Z [i, j] | + | Z [i, j + 2] -Z [i, j] |) / 2 158
Horizontal RR (BB) similarity component:
Ch3 [i, j] = (| Z [i-3, j] -Z [i, j] | +3 ・ | Z [i + 1, j] -Z [i, j] |) / 4 ・ ・Formula 159
Vertical GR (GB) similarity component:
Cv4 [i, j] = (| G [i, j-1] -Z [i, j] | + | G [i, j + 11-Z [i, j] |) / 2 Equation 160
Horizontal GR (GB) similarity component:
Gh4 [i, j] = (2 ・ | G [i-1, j] -Z [i, j] | + | G [i + 2, j] -Z [i, j] |) / 3 Formula 161
Vertical BG (RG) similarity component:
Cv5 [i, j] = (| Z [i-1, j-1] -G [i-1, j] | + | Z [i-1, j + 1] -G [i-1, j] |
+ | Z [i + 2, j-1] -G [i + 2, j] | + | Z [i + 2, j + 1] -G [i + 2, j] |) / 4 Formula 162
Horizontal BG (RG) similarity component:
Ch5 [i, j] = (2 ・ | Z [i-1, j-1] -G [i, j-1] | +2 ・ | Z [i-1, j + 1] -G [i, j + 1] |
+ | Z [i + 2, j-1] -G [i, j-1] | + | Z [i + 2, j + 1] -G [i, j + 1] |) / 6 Formula 163
Vertical luminance similarity component:
Cv6 [i, j] = (| Y [i, j-1] -Y [i, j] | + | Y [i, j + 1] -Y [i, j] |) / 2 164
Lateral luminance similarity component:
Ch6 [i, j] = (2 ・ | Y [i-1, j] -Y [i, j] | + | Y [i + 2, j] -Y [i, j] |) / 3 ・ ・・ Formula 165
However, in Equation 164 and Equation 165, Y [i, j] is a value calculated by Equation 166, and the color information of the color component of the sub-pixel located near the interpolation target sub-pixel is R: G: B This corresponds to the luminance generated by the filtering process that averages at a ratio of 1: 2: 1.
[0228]
Y [i, j] = (4 ・ A [i, j] +2 ・ (A [i, j-1] + A [i, j + 1] + A [i-1, j] + A [i + 2, j])
+ A [i-1, j-1] + A [i-1.j + 1] + A [i + 2, j-11 + A [i + 2, j + 1]) / 16 166
However, in Expression 166, A [i, j] represents arbitrary color information on the sub-pixel type Bayer array, and takes a value of G or Z depending on the arrangement location.
In addition, although the element of the similarity which comprises each similarity component mentioned above is calculated using the absolute value of a difference, you may calculate by the square of an absolute value, a power, etc.
[0229]
Next, the interpolation processing unit 17 is similar to Expression 126 and Expression 127 of the fourth embodiment (using weighting coefficients that play the same role as the weighting coefficients of Expression 23 and Expression 24 of the first embodiment). Then, Cv0 [i, j] and Ch0 [i, j] are calculated by weight-adding a plurality of types of similarity components for each direction.
Cv0 [i, j] and Ch0 [i, j] calculated in this way are directly used as the vertical similarity Cv [i, j] and the horizontal similarity Ch [i, j] of the interpolation target subpixel. j]. Here, an example is shown in which these similarities are calculated by peripheral addition.
[0230]
That is, the interpolation processing unit 17 obtains values (Cv0 [i, j], Cv0 [i + 1, j], Cv0 obtained by weighted addition for each direction of similarity components in the subpixels to be interpolated and the peripheral subpixels. [i-1, j-1], Cv0 [i-1, j + 1], Cv0 [i + 2, j-1], Cv0 [i + 2, j + 1], etc.) Weighted addition as in Addition Method 1 >> or << Peripheral Addition Method 2 >> to calculate the vertical similarity Cv [i, j] and the horizontal similarity Ch [i, j] of the interpolation target subpixel. .
[0231]
<< Peripheral addition method 1 >>
Cv [i, j] = (2 ・ (Cv0 [i, j] + Cv0 [i + 1, j]) + Cv0 [i-1, j-1] + Cv0 [i-1, j + 1]
+ Cv0 [i + 2, j-1] + Cv0 [i + 2, j + 1]) / 8 ・ ・ ・ Equation 167
Ch [i, j] = (2 ・ (Ch0 [i, j] + Ch0 [i + 1, j]) + Ch0 [i-1, j-1] + Ch0 [i-1, j + 1]
+ Ch0 [i + 2, j-1] + Ch0 [i + 2, j + 1]) / 8 ・ ・ ・ Equation 168
<< Peripheral addition method 2 >>
Cv [i, j] = (2 ・ (Cv0 [i, j] + Cv0 [i + 1, j])
+2 ・ (Cv0 [i-1, j-1] + Cv0 [i-1, j + 1] + Cv0 [i + 2, j-1] + Cv0 [i + 2, j + 1])
+ (Cv0 [i, j-2] + Cv0 [i + 1, j-2]) / 2
+ (Cv0 [i, j + 2] + Cv0 [i + 1, j + 2]) / 2
+ Cv0 [i-3, j] + Cv0 [i + 4, j]) / 16 ・ ・ ・ Equation 169
Ch [i, j] = (2 ・ (Ch0 [i, j] + Ch0 [i + 1, j])
+2 ・ (Ch0 [i-1, j-1] + Ch0 [i-1, j + 1] + Ch0 [i + 2, j-1] + Ch0 [i + 2, j + 1])
+ (Ch0 [i, j-2] + Ch0 [i + 1, j-2]) / 2
+ (Ch0 [i, j + 2] + Ch0 [i + 1, j + 2]) / 2
+ Ch0 [i-3, j] + Ch0 [i + 4, j]) / 16 ・ ・ ・ Equation 170
Note that the position of the color information used when calculating the similarity of the subpixel located at the coordinates [i, j] and the subpixel located at the coordinates [i + 1, j] (on the left side of the pixel on the Bayer array) The position of the color information used when calculating the similarity of the subpixels) is the subpixel of the i-th column and the subpixel of the i + 1-th column of the subpixel-type Bayer array in FIG. If the straight line passing between them is symmetrical, the vertical similarity Cv [i + 1, j] and the horizontal similarity Ch [i + 1] of the subpixel located at the coordinates [i + 1, j] , j] can be calculated by replacing i−3 to i + 4 described in Equations 154 to 170 as follows.
[0232]
i-3 → i + 4, i-2 → i + 3, i-1 → i + 2, i → i + 1, i + 1 → i, i + 2 → i-1, i + 3 → i- 2, i + 4 → i-3.
When the interpolation processing unit 17 calculates the vertical and horizontal similarities as described above, based on these similarities, similarities in the vertical and horizontal directions in the interpolation target subpixel (hereinafter referred to as “vertical and horizontal similarities”). (Referred to as “sex”) (S2 in FIG. 25). Then, the interpolation processing unit 17 uses the index HV [i as the vertical / horizontal main similarity determination result in the first embodiment._p, j], a value is set in the index HV [i, j] as a result of determining the vertical / horizontal similarity in the same manner as the process in which the value is set in (j) (S25 to S5 in FIG. 25).
[0233]
Note that the setting of the index HV [i + 1, j] indicating the vertical and horizontal similarity of the subpixel located at the coordinates [i + 1, j] is performed by the vertical similarity Cv [i calculated as described above. By using + 1, j] and the horizontal similarity Ch [i + 1, j], it can be set in the same manner as the index HV [i, j].
Next, the interpolation processing unit 17 determines what value the index HV [i, j] is (S6 in FIG. 25), and the green of the subpixel to be interpolated located at the coordinates [i, j] as follows. The interpolation value G [i, j] is calculated.
[0234]
When HV [i, j] = 1 G [i, j] = GvL: FIG. 25S7
When HV [i, j] = 0, G [i, j] = (GvL + GhL) / 2: FIG. 25S8
When HV [i, j] = 1 G [i, j] = GhL: FIG. 25S9
However, GvL and GhL are values calculated as in Expression 41, Expression 43, Expression 45, and Expression 47 of the first embodiment.
[0235]
Note that the green interpolation value G [i + 1, j] of the interpolation target subpixel located at the coordinates [i + 1, j] is any index HV [i + 1, j] set as described above. By determining whether it is a value or not, it can be calculated as follows.
When HV [i + 1, j] = 1, G [i + 1, j] = GvR
When HV [i + 1, j] = 0, G [i + 1, j] = (GvR + GhR) / 2
When HV [i + 1, j] = 1, G [i + 1, j] = GhR
However, GvR and GhR are values calculated as in Expression 42, Expression 44, Expression 46, and Expression 48 of the first embodiment.
[0236]
As described above, in the fifth embodiment, the similarity can be calculated for each interpolation target sub-pixel, and the interpolation value can be calculated according to the similarity determination result based on the similarity.
In the fifth embodiment, a plurality of types of similarity components defined by Expressions 154 to 166 are weighted by weighting coefficients that play the same role as the weighting coefficients of Expressions 23 and 24 in the first embodiment. Addition is performed to calculate the vertical similarity Cv [i, j] and the horizontal similarity Ch [i, j] of the sub-pixel to be interpolated, and the index HV [i, j based on such similarity However, the setting of the index HV [i, j] can also be realized by the following processing (corresponding to claims 45 to 52).
[0237]
First, the interpolation processing unit 17 calculates a plurality of types of similarity components defined by Equations 154 to 163, and these similarity components are the same as the weighting coefficients of Equations 89 to 92 in the third embodiment. Weighted by the weighting coefficient that plays a role, the same-color similarity CvD [i, j] in the vertical direction and the same-color similarity ChD [i, j] in the horizontal direction, and the different colors in the vertical direction The similarity CvN [i, j] and the horizontal dissimilarity similarity ChN [i, j] are calculated. In the process of calculating these similarities, it is assumed that marginal addition is performed in the same manner as Expressions 167 to 170.
[0238]
Next, the interpolation processing unit 17 is similar to Condition 1 of the third embodiment for the vertical different-color similarity CvN [i, j] and the horizontal different-color similarity ChN [i, j]. It is determined whether the image in the vicinity of the interpolation target subpixel is an achromatic portion or a chromatic portion.
Then, when the image near the sub-pixel to be interpolated is an achromatic portion, the interpolation processing unit 17 performs the different-color similarity CvN [i, j] in the vertical direction and the different-color similarity ChN [i, j in the horizontal direction. If the image HV [i, j] is set to the index HV [i, j] and the image near the sub-pixel to be interpolated is a chromatic part, the similarity between the same colors in the vertical direction CvD [i , j] and the same-color similarity degree ChD [i, j] in the horizontal direction, the vertical and horizontal similarity is determined and a value is set for the index HV [i, j]. Note that such vertical / horizontal similarity determination and index HV [i, j] setting processing are performed using the index HV [i as the vertical / horizontal main similarity determination result in the first embodiment._p, j] is performed in the same manner as the processing in which a value is set.
[0239]
That is, the index HV [i, j] set as described above includes the index HV [i] of the third embodiment._p, j], the vertical / horizontal similarity determined based on the similarity suitable for each portion of the achromatic portion and the chromatic portion is reflected.
<< Sixth Embodiment >>
The operation of the sixth embodiment will be described below. Note that the RB interpolation processing in the sixth embodiment can be performed in the same manner as in the first embodiment, and thus description thereof is omitted here.
[0240]
FIG. 26 is an operation flowchart of the interpolation processing unit 17 in the sixth embodiment, and shows the operation of the interpolation processing unit 17 in the G interpolation processing. However, FIG. 26 shows an example in which a subpixel located on the left side of a pixel on the Bayer array is set as an interpolation target.
Hereinafter, although the G interpolation process in the sixth embodiment will be described, detailed description of the same operation as that of the fifth embodiment will be omitted.
[0241]
First, as in the fifth embodiment, the interpolation processing unit 17 calculates the vertical and horizontal similarities of the interpolation target sub-pixels (FIG. 26S1), and compares the vertical and horizontal main similarities (FIG. 26S2). Then, a value is set in the index HV [i, j] (FIGS. 26S3 to S5).
Next, the interpolation processing unit 17 calculates the similarity in the oblique 45 degree direction and the oblique 135 degree direction in the interpolation target sub-pixel (S6 in FIG. 26).
[0242]
Here, details of the process of calculating the similarity C45 [i, j] in the oblique 45 ° direction and the similarity C135 [i, j] in the oblique 135 ° direction in the subpixel to be interpolated located at the coordinates [i, j] Will be explained.
The interpolation processing unit 17 calculates a plurality of types of similarity components for the oblique 45 degree direction and the oblique 135 degree direction defined by Expressions 171 to 178.
[0243]
The similarity component between GG in the 45-degree direction:
C45_1 [i, j] = (| G [i-2, j] -G [i, j-1] | + | G [i-1, j] -G [i + 1, j-1] |
+ | G [i, j + 11-G [i + 2, j] | + | G [i + 1, j + 1] -G [i + 3, j] |) / 4 Equation 171
GG similarity component in the direction of diagonal 135 degrees:
C135_1 [i, j] = (| G [i-2, j] -G [i, j + 1] | + | G [i-1, j] -G [i + 1, j + 1] |
+ | G [i, j-1] -G [i + 2, j] | + | G [i + 1, j-1] -G [i + 3, j] |) / 4 Equation 172
BB (RR) similarity component in a 45-degree direction:
C45_2 [i, j] = (| Z [i-2, j + 1] -Z [i + 2, j-1] |
+ | Z [i-1, j + 1] -Z [i + 3, j-1] |) / 2
BB (RR) similarity component in the direction of diagonal 135 degrees:
C135_2 [i, j] = (| Z [i-2, j-1] -Z [i + 2, j + 1] |
+ | Z [i-1, j-1] -Z [i + 3, j + 1] |) / 2
RR (BB) similarity component in the 45 ° direction:
C45_3 [i, j] = (| Z [i-4, j + 2] -Z [i, j] | + | Z [i-3, j + 2] -Z [i + 1, j] |
+ | Z [i + 4, j-2] -Z [i, j] | + | Z [i + 5, j-2] -Z [i + 1, j] |) / 4 Equation 175
RR (BB) similarity component in the direction of diagonal 135 degrees:
C135_3 [i, j] = (| Z [i-4, j-2] -Z [i, j] | + | Z [i-3, j-2] -Z [i + 1, j] |
+ | Z [i + 4, j + 2] -Z [i, j] | + | Z [i + 5, j + 2] -Z [i + 1, j] |) / 4 Equation 176
BR (RB) similarity component in 45 degree direction:
C45_4 [i, j] = (| Z [i-2, j + 1] -Z [i, j] | + | Z [i-1, j + 1] -Z [i + 1, j] |
+ | Z [i + 2, j-1] -Z [i, j] | + | Z [i + 3, j-1] -Z [i + 1, j] |) / 4
BR (RB) similarity component in diagonal direction of 135 degrees:
C135_4 [i, j] = (| Z [i-2, j-1] -Z [i, j] | + | Z [i-1, j-1] -Z [i + 1, j] |
+ | Z [i + 2, j + 1] -Z [i, j] | + | Z [i + 3, j + 1] -Z [i + 1, j] |) / 4
In addition, although the element of the similarity which comprises each similarity component mentioned above is calculated using the absolute value of a difference, you may calculate by the square of an absolute value, a power, etc.
[0244]
Next, as shown in Expression 179 and Expression 180, the interpolation processing unit 17 uses the weighting coefficients b1, b2, b3, and b4, which play the same role as in the second embodiment, to each of a plurality of types of similarity components. Weighted addition for each direction.
C45_0 [i, j] = (b1, C45_1 [i, j] + b2, C45_2 [i, j] + b3, C45_3 [i, j]
+ b4 ・ C45_4 [i, j]) / (b1 + b2 + b3 + b4) ・ ・ ・ Equation 179
C135_0 [i, j] = (b1, C135_1 [i, j] + b2, C135_2 [i_p, j] + b3 ・ C135_3 [i, j]
+ b4 ・ C135_4 [i, j]) / (b1 + b2 + b3 + b4) ・ ・ ・ Formula 180
C45_0 [i, j] and C135_0 [i, j] calculated in this way are assumed to be the similarities C45 [i, j] and C135 [i, j] in the diagonal direction of the sub-pixels to be interpolated as they are. Here, an example is shown in which these similarities are calculated by peripheral addition.
[0245]
That is, the interpolation processing unit 17 obtains values (C45_0 [i, j], C45_0 [i + 1, j], C45_0) obtained by weighted addition for each direction of similarity components in the subpixels to be interpolated and the peripheral subpixels. [i-1, j-1], C45_0 [i-1, j + 1], C45_0 [i + 2, j-1], C45_0 [i + 2, j + 1], etc.) The weighted addition is performed as in addition method 1 >> or << peripheral addition method 2 >> to calculate the similarity C45 [i, j], C135 [i, j] in the diagonal direction of the interpolation target subpixel.
[0246]
<< Peripheral addition method 1 >>
C45 [i, j] = (2 ・ (C45_0 [i, j] + C45_0 [i + 1, j]) + C45_0 [i-1, j-1] + C45_0 [i-1, j + 1]
+ C45_0 [i + 2, j-1] + C45_0 [i + 2, j + 1]) / 8 ・ ・ ・ Formula 181
C135 [i, j] = (2 ・ (C135_0 [i, j] + C135_0 [i + 1, j]) + C135_0 [i-1, j-1]
+ C135_0 [i-1, j + 1] + C135_0 [i + 2, j-1] + C135_0 [i + 2, j + 1]) / 8 ... Formula 182
<< Peripheral addition method 2 >>
C45 [i, j] = (2 ・ (C45_0 [i, j] + C45_0 [i + 1, j])
+2 ・ (C45_0 [i-1, j-1] + C45_0 [i-1, j + 1] + C45_0 [i + 2, j-1] + C45_0 [i + 2, j + 1])
+ (C45_0 [i, j-2] + C45_0 [i + 1, j-2]) / 2
+ (C45_0 [i, j + 2] + C45_0 [i + 1, j + 2]) / 2
+ C45_0 [i-3, j] + C45_0 [i + 4, j]) / 16 ... Formula 183
C135 [i, j] = (2 ・ (C135_0 [i, j] + C135_0 [i + 1, j])
+2 ・ (C135_0 [i-1, j-1] + C135_0 [i-1, j + 1] + C135_0 [i + 2, j-1] + C135_0 [i + 2, j + 1]
+ (C135_0 [i, j-2] + C135_0 [i + 1, j-2]) / 2
+ (C135_0 [i, j + 2] + C135_0 [i + 1, j + 2]) / 2
+ C135_0 [i-3, j] + C135_0 [i + 4, j]) / 16 ... Formula 184
Note that the position of the color information used when calculating the similarity of the subpixel located at the coordinates [i, j] and the subpixel located at the coordinates [i + 1, j] (on the left side of the pixel on the Bayer array) The position of the color information used in calculating the similarity of the subpixels) is the subpixel of the i-th column and the subpixel of the i + 1-th column of the subpixel-type Bayer array in FIG. If the straight line passing between them is symmetrical, the subpixel located at the coordinates [i + 1, j] has a similarity C45 [i + 1, j] in the oblique 45 ° direction and a similarity C135 in the oblique 135 ° direction. [i + 1, j] can be calculated by replacing i-3 to i + 4 described in Equations 171 to 184 as follows.
[0247]
i-3 → i + 4, i-2 → i + 3, i-1 → i + 2, i → i + 1, i + 1 → i, i + 2 → i-1, i + 3 → i- 2, i + 4 → i-3.
When the interpolation processing unit 17 calculates the similarity in the oblique direction as described above, the similarity with respect to the oblique direction in the interpolation target subpixel (hereinafter referred to as “oblique similarity”) is calculated based on the similarity. Determination is made (S7 in FIG. 26). Then, the interpolation processing unit 17 uses the index DN [i as the determination result of the diagonal main similarity in the second embodiment._p, j], a value is set in the index DN [i, j] as a result of the diagonal similarity determination (S8 to S10 in FIG. 26).
[0248]
Note that the setting of the index DN [i + 1, j] indicating the diagonal similarity of the sub-pixel located at the coordinates [i + 1, j] is performed by the similarity C45 in the diagonal 45 ° direction calculated as described above. By using [i + 1, j] and the degree of similarity C135 [i + 1, j] in the oblique 135-degree direction, it can be set in the same manner as the index DN [i, j].
Next, the interpolation processing unit 17 determines what values the index HV [i, j] and the index DN [i, j] are (S11 in FIG. 26), and the similarity of the interpolation target pixels in the second embodiment. Similar to the processing in which the strength of the sub-pixels is classified into case1 to case9, the similarity strength of the subpixels to be interpolated is classified into any of case1 to case9.
[0249]
Then, the interpolation processing unit 17 calculates the green interpolation value G [i, j] of the interpolation target subpixel located at the coordinates [i, j] as follows according to the result of such classification. .
In case 1: G [i, j] = Gv45L: FIG. 26S12
In case 2: G [i, j] = GvL: FIG. 26S13
For case 3: G [i, j] = Gv135L: FIG.
For case 4: G [i, j] = (Gv45L + Gh45L) / 2: FIG. 26S15
For case 5: G [i, j] = (GvL + GhL) / 2: FIG. 26 S16
In case 6: G [i, j] = (Gv135L + Gh135L) / 2: FIG. 26S17
In case 7: G [i, j] = Gh45L: FIG. 26 S18
In case 8: G [i, j] = GhL: FIG. 26S19
In case 9: G [i, j] = Gh135L: FIG. 26S20
However, GvL, Gv45L, Gv135L, GhL, Gh45L, and Gh135L are values calculated as in Expression 77 to Expression 82 of the second embodiment.
[0250]
The green interpolation value G [i + 1, j] of the interpolation target subpixel located at the coordinates [i + 1, j] is the index HV [i + 1, j] and the index set as described above. By determining what value DN [i, j] is and classifying the strength of similarity of the sub-pixels to be interpolated into any of case1 to case9, it can be calculated as follows.
For case1: G [i + 1, j] = Gv45R
Case2: G [i + 1, j] = GvR
Case3: G [i + 1, j] = Gv135R
Case4: G [i + 1, j] = (Gv45R + Gh45R) / 2
For case5: G [i + 1, j] = (GvR + GhR) / 2
For case6: G [i + 1, j] = (Gv135R + Gh135R) / 2
For case7: G [i + 1, j] = Gh45R
Case8: G [i + 1, j] = GhR
Case9: G [i + 1, j] = Gh135R
However, GvR, Gv45R, Gv135R, GhR, Gh45R, and Gh135R are values calculated as in Expression 83 to Expression 88 of the second embodiment.
[0251]
As described above, in the sixth embodiment, the similarity can be calculated for each interpolation target sub-pixel, and the interpolation value can be calculated according to the similarity determination result based on the similarity.
Note that the similarities C45 [i, j] and C45 [i + 1, j] in the oblique 45 ° direction and the similarities C135 [i, j] and C135 [i + 1, j] in the oblique 135 ° direction are expressed by equations It may be calculated by 185 and 186.
[0252]
C45 [i, j] = C45 [i + 1, j] =
((| R [i + 4, j-2] -R [i, j] |-| R [i-4, j + 2] -R [i, j] |) / 2
+ (| R [i + 5, j-2] -R [i + 1, j] |-| R [i-3, j + 2] -R [i + 1, j] |) / 2) / 2 ... Formula 185
C135 [i, j] = C135 [i + 1, j] =
((| R [i-4, j-2] -R [i, j] |-| R [i + 4, j + 2] -R [i, j] |) / 2
+ (| R [i-3, j-2] -R [i + 1, j] |-| R [i + 5, j + 2] -R [i + 1, j] |) / 2) / 2 ・ ・ ・ Formula 186
By the way, the above-described interpolation processing of the first to sixth embodiments is not limited to the sub-pixel type Bayer array composed of three colors, but is applied to a filter array of three or more colors having a sub-pixel structure. Is also possible.
[0253]
<< Seventh Embodiment >>
The operation of the seventh embodiment will be described below.
FIG. 27 is a functional block diagram of the seventh embodiment.
Note that the seventh embodiment corresponds to executing interpolation processing by a personal computer using a recording medium on which the interpolation processing program according to claims 59 to 67 is recorded.
[0254]
27, components having the same functions as those in the functional block diagram shown in FIG. 1 are denoted by the same reference numerals, and description of the configuration is omitted.
27 is different from the electronic camera 10 shown in FIG. 1 in that the control unit 21 and the image processing unit 22 in FIG. 27 are different from the control unit 11 and the image processing unit 15 in FIG. The interface unit 23 shown in FIG. 27 is newly provided.
[0255]
In FIG. 27, the personal computer 30 includes a CPU 31, an interface unit 32, a hard disk 33, and a memory 34. The CPU 31 is connected to the interface unit 32, the hard disk 33, and the memory 34 via a bus.
Note that the personal computer 30 is preinstalled with an interpolation processing program (an interpolation processing program that executes interpolation processing in the same manner as the interpolation processing unit 17 of each embodiment described above) recorded on a recording medium such as a CD-ROM. It shall be. That is, the hard disk 33 stores such an interpolation processing program in an executable state.
[0256]
The operation of the seventh embodiment will be described below with reference to FIG.
First, in the electronic camera 20, image data generated in the same manner as the electronic camera 10 illustrated in FIG. 1 is supplied to the image processing unit 22. The image processing unit 22 performs image processing (for example, gradation conversion processing) other than the interpolation processing on the image data, and the recording unit 16 records the image data subjected to the image processing in the form of an image file.
[0257]
Such an image file is supplied to the personal computer 30 via the interface unit 23.
When the CPU 31 in the personal computer 30 acquires the image file via the interface unit 32, the CPU 31 executes the above-described interpolation processing program. The image data in which the resolution of each color component is increased by the interpolation processing is subjected to image compression or the like as necessary, and then recorded on the hard disk 33 or the like, and finally displayed in accordance with each connected device such as a display and a printer. Output as color system data.
[0258]
That is, in the seventh embodiment, the personal computer 30 can perform the same interpolation processing as that of the above-described embodiments.
[0259]
【The invention's effect】
  As mentioned above1st, 27thIn this invention, it is possible to accurately determine the strength of rough similarity at the pixel pitch,2nd, 3rd and 5thIn this invention, it is possible to determine the level of rough similarity at the pixel pitch and to determine the level of fine similarity at the sub-pixel pitch. Also,4th to 6th, 28thIn this invention, the strength of similarity is determined using not only color information located at a target distance but also color information located at an asymmetric distance for the interpolation target subpixel. further,If you determine the strength of the similarity based on the similarity,The degree of similarity can be accurately determined, and the interpolation value of the interpolation target subpixel can be calculated with high accuracy.
[0260]
  7thIn the invention, even if it is determined that the direction of strong similarity is unknown as a result of the determination by the main similarity determination means, even if the similarity strength is determined by the fine part similarity determination means, The accuracy of strength determination does not decrease, and it is similar compared to the case where the strength of similarity is determined for all subpixels by the main similarity determination means and the fine part similarity determination means. It is possible to quickly determine the strength of the property and to quickly calculate the interpolation value of the interpolation target subpixel.
[0261]
  8th to 10thIn the invention, since the similarity is determined using the color information of a plurality of color components, the similarity can be obtained even for an image whose color tone changes (an image where the ratio of color information changes between different color components). Can be accurately determined. In these inventions, the color information of a plurality of color components can be used as the color information of the sub-pixel to be interpolated and the color information existing in the vicinity thereof, so that the strength of the similarity can be determined. Even if the image is high, similarity can be determined accurately. Therefore, according to these inventions, even when the image to be subjected to the interpolation process includes a “part where the color tone changes” or a “part where the spatial frequency is high”, the entire image is kept in harmony with high accuracy. Interpolation processing can be performed.
[0262]
  11th and 23rdThe invention uses different similarity between different colors and similarity between different colors according to the characteristics of the image in the local region to which the interpolation target sub-pixel belongs, or performs weighted addition between different color similarities and similarity between the same colors Since the weight ratio is changed, it is possible to calculate the degree of similarity suitable for determining the degree of similarity, and by calculating the similarity between different colors and the similarity between the same color using a plurality of color information The calculation accuracy of can be increased. Therefore, the interpolation amount can be calculated with high accuracy.
[0263]
  12th, 24thIn this invention, the similarity between different colors is calculated by using color information existing at a distance interval shorter than the similarity between the same colors, so that the similarity between images having a high spatial frequency is reflected in the similarity between different colors. That is, when determining the strength of similarity using the similarity between different colors, or extracting the image features in the local region to which the interpolation target sub-pixel belongs using the similarity between different colors as information about the color, The resolution capability of the high frequency part can be effectively extracted. As a result, the accuracy of similarity determination can be increased, and the accuracy of the interpolation amount can be increased.
[0264]
  13thAccording to the invention, the similarity between different colors and the similarity component using many color components for the calculation of the similarity between the same colors can be generated, so that the similarity can be calculated with high accuracy and the accuracy of the interpolation amount is improved. It is done.
  14thSince the invention extracts the image features in the local region to which the interpolation target sub-pixel belongs by using the information on the color, it is possible to selectively use the similarity between the different colors and the similarity between the same colors, When setting the weight ratio with the inter-similarity, image features can be extracted based on an appropriate index. Therefore, the similarity suitable for determining the strength of similarity can be calculated, and the accuracy of the interpolation amount can be increased.
[0265]
  15th and 16thAccording to the invention, when extracting image features in a local region to which the interpolation target subpixel belongs, the similarity between different colors is used as information about the color, so the similarity between different colors is calculated to determine the strength of similarity. If it does, it is not necessary to ask for information about color again. That is, the entire interpolation process can be performed efficiently.
[0266]
  17th and 25thIn the invention, it is possible to calculate the similarity of the interpolation target subpixel while considering the continuity in the direction in which the similarity is strong between the interpolation target subpixel and the surrounding subpixels.18th, 26thIn the invention, it is possible to reduce erroneous determination of similarity caused by noise or the like. That is, in these inventions, the accuracy of similarity determination can be increased, so that the accuracy of the interpolation amount is increased.
[0267]
  19thIn the invention, the color information of one specific color component or the color information of a plurality of types of color components can be reflected in the similarity in at least two directions in the sub-pixel to be interpolated,20thIn this invention, the color information of a plurality of types of color components can be reflected in the similarity in at least two directions in the sub-pixel to be interpolated. Also,21stIn the invention, color information of a plurality of types of color components can be accurately reflected in the similarity in at least two directions in the sub-pixel to be interpolated,22ndIn the invention, an appropriate similarity can be calculated without being affected by the difference in the shortest distance with respect to each similarity component. Therefore, according to these inventions, it is possible to determine the strength of similarity while reflecting the characteristics of the image of the local region including the sub-pixel to be interpolated, so that the interpolation value is calculated without breaking the harmony of the entire image. can do.
[Brief description of the drawings]
FIG. 1 is a functional block diagram of an electronic camera corresponding to the first to sixth embodiments.
FIG. 2 is an operation flowchart (1) of an interpolation processing unit in the first embodiment.
FIG. 3 is an operation flowchart (2) of an interpolation processing unit in the first embodiment.
FIG. 4 is a diagram illustrating a peripheral addition method.
FIG. 5 is a diagram illustrating a position of color information used when calculating a fine similarity component.
FIG. 6 is a diagram illustrating the position of color information used in color difference interpolation.
FIG. 7 is a diagram illustrating linear interpolation of color differences.
FIG. 8 is a diagram illustrating a process of interpolating a color difference with a median value.
FIG. 9 is a diagram for explaining the range of median processing;
FIG. 10 is a diagram for explaining the influence of lateral chromatic aberration.
FIG. 11 is a diagram for explaining overcorrection due to chromatic aberration of magnification.
FIG. 12 is an operation flowchart of an interpolation processing unit in the second embodiment.
FIG. 13 is an operation flowchart (continued) of the interpolation processing unit in the second embodiment.
[Fig.14] (HV [i_p, j], DN [i_p, j]) is a diagram showing a direction of strong similarity corresponding to the value.
FIG. 15 is a diagram (1) illustrating a position of color information used when calculating a green interpolation value;
FIG. 16 is a diagram (2) illustrating a position of color information used when calculating a green interpolation value;
FIG. 17 is a diagram (3) illustrating a position of color information used when a green interpolation value is calculated.
FIG. 18 is a diagram for explaining how the influence of lateral chromatic aberration is eliminated.
FIG. 19 is an operation flowchart (1) of an interpolation processing unit in the third embodiment.
FIG. 20 is an operation flowchart (2) of the interpolation processing unit in the third embodiment.
FIG. 21 is an operation flowchart (3) of the interpolation processing unit in the third embodiment.
FIG. 22 is a diagram illustrating a relationship between similarity and strength of similarity.
FIG. 23 is a diagram illustrating a filter for generating luminance.
FIG. 24 is a diagram illustrating a peripheral addition method in a sub-pixel type Bayer array.
FIG. 25 is an operation flowchart of the interpolation processing unit in the fifth embodiment.
FIG. 26 is an operation flowchart of the interpolation processing unit in the sixth embodiment.
FIG. 27 is a functional block diagram according to a seventh embodiment.
FIG. 28 is a diagram illustrating an arrangement of color components in the image sensor.
[Explanation of symbols]
10, 20 Electronic camera
11, 21 Control unit
12 Shooting optical system
13 Imaging unit
14 A / D converter
15, 22 Image processing unit
16 Recording section
17 Interpolation processing unit
23, 32 Interface section
30 Personal computer
31 CPU
33 Hard disk
34 memory

Claims (28)

One color component is associated with each pixel of a plurality of two-dimensionally arranged pixels, and image data generated by an imaging device in which each pixel is divided into two subpixels is assigned to each subpixel. In an interpolation processing apparatus that performs processing for determining an interpolation value corresponding to color information of a missing color component,
A value obtained by grouping color information output from two subpixels constituting each pixel is set as pixel output color information corresponding to the color information of the pixel, and the similarity of the interpolation target subpixels in at least two directions is strong or weak Main similarity determination means for determining the image using the pixel output color information;
In accordance with the strength of the similarity determined by the main similarity determination means, color information of the subpixel used when calculating the interpolation value of the interpolation target subpixel is selected, and the interpolation target subpixel is selected using the color information. Interpolation value calculation means for calculating the interpolation value of the interpolation processing apparatus. The interpolation processing device according to claim 1,
Fine portion similarity determination means for determining, using the subpixel output color information, the strength of similarity of the interpolation target subpixel in at least two directions, using the color information output from the subpixel as subpixel output color information. Prepared,
The interpolation value calculating means includes
According to the similarity determined by the main similarity determination unit and the fine part similarity determination unit, color information of the subpixel used when calculating the interpolation value of the interpolation target subpixel is selected, An interpolation processing apparatus that calculates an interpolation value of an interpolation target sub-pixel using color information. The interpolation processing device according to claim 2,
The main similarity determination means includes
Using the pixel output color information of the pixel located at a symmetrical distance with respect to the pixel to which the interpolation target subpixel belongs, determining the strength of the similarity,
The fine part similarity determination means includes:
An interpolation processing apparatus characterized in that the strength of the similarity is determined using subpixel subpixel output color information located at an asymmetric distance with respect to the interpolation target subpixel. One color component is associated with each pixel of a plurality of two-dimensionally arranged pixels, and image data generated by an imaging device in which each pixel is divided into two subpixels is assigned to each subpixel. In an interpolation processing apparatus that performs processing for determining an interpolation value corresponding to color information of a missing color component,
Primary similarity determination means for determining the strength of the similarity of the interpolation target subpixel in at least two directions using color information located at a symmetrical distance with respect to the interpolation target subpixel;
Fine part similarity determination means for determining the strength of the similarity of the interpolation target subpixel in at least two directions using color information located at an asymmetric distance with respect to the interpolation target subpixel;
According to the similarity determined by the main similarity determination unit and the fine part similarity determination unit, color information of the subpixel used when calculating the interpolation value of the interpolation target subpixel is selected, An interpolation processing apparatus comprising: interpolation value calculation means for calculating an interpolation value of a sub-pixel to be interpolated using color information. The interpolation processing device according to claim 4, wherein
The main similarity determination means includes
As the color information located at the target distance, a value obtained by grouping color information output from two subpixels constituting a pixel located at the target distance with respect to the pixel to which the interpolation target subpixel belongs is used.
The fine part similarity determination means includes:
As the color information located at the asymmetric distance, color information output from the subpixel located at an asymmetric distance with respect to the interpolation target subpixel is used. The interpolation processing device according to claim 4, wherein
The main similarity determination means includes
As the color information located at the target distance, color information output from the subpixel located at a symmetric distance with respect to the interpolation target subpixel,
The fine part similarity determination means includes:
As the color information located at the asymmetric distance, color information output from the subpixel located at an asymmetric distance with respect to the interpolation target subpixel is used. In the interpolation processing device according to any one of claims 2 to 6,
The fine part similarity determination means includes:
An interpolation processing apparatus, wherein when the direction of strong similarity is determined to be unknown as a result of the determination by the main similarity determination means, the similarity is determined. The interpolation processing apparatus according to any one of claims 1 to 7, wherein the image data is represented by a color system composed of a plurality of color components.
The main similarity determination means includes
Calculating the similarity in at least two directions in the interpolation target sub-pixel using color information of a plurality of color components, and determining the strength of the similarity of the interpolation target sub-pixel in at least two directions based on the similarity. An interpolation processing device characterized by the above. 9. The interpolation processing apparatus according to claim 8, wherein the image data is represented by a color system composed of first to third color components, and the first color component is compared with the second color component and the third color component. When determining an interpolation value corresponding to the color information of the first color component for a sub-pixel having a high spatial frequency and lacking the first color component,
The main similarity determination means includes
(1) a similarity component using color information of the first color component and color information of the second color component;
(2) a similarity component using color information of the first color component and color information of the third color component;
(3) a similarity component using only color information of the first color component;
(4) a similarity component using only the color information of the second color component;
(5) Among the plurality of similarity components with the similarity component using only the color information of the third color component, the combination including the color information of the first to third color components is used, and the interpolation target sub An interpolation processing device characterized by calculating a similarity in at least two directions in a pixel. The interpolation processing device according to claim 9, wherein
The main similarity determination means includes
An interpolation processing apparatus, wherein among the plurality of similarity components, similarity components belonging to the combination are weighted and added for each direction to calculate the similarity in at least two directions in the interpolation target subpixel. The interpolation processing apparatus according to any one of claims 1 to 7, wherein the image data is represented by a color system composed of a plurality of color components.
The main similarity determination means includes
Based on a plurality of color information selected from the color information of the interpolation target sub-pixel and the color information existing in the vicinity thereof,
(1) The similarity between different colors calculated using color information of different color components;
(2) Calculate two types of similarity with the same-color similarity calculated using the color information of the same color component, and either one according to the feature of the image in the local region to which the interpolation target subpixel belongs By switching to one of the similarities, or by switching the weighting ratio when each similarity is weighted and added, the similarity for at least two directions in the subpixel to be interpolated is calculated, and based on the similarity, at least for one direction An interpolation processing device characterized by determining the strength of similarity of interpolation target sub-pixels. The interpolation processing device according to claim 11,
The main similarity determination means includes
An interpolation processing apparatus, wherein the similarity between different colors is calculated using color information existing at a distance interval shorter than the similarity between the same colors. 13. The interpolation processing apparatus according to claim 11, wherein the image data is represented by a color system composed of first to third color components, and the first color component is a second color component and a third color. When determining an interpolation value corresponding to the color information of the first color component for a sub-pixel having a higher spatial frequency than the component and lacking the first color component,
The main similarity determination means includes
As the similarity between the different colors,
(1) A different color similarity component composed of color information of the first color component and color information of the second color component;
(2) calculating a similarity including at least one of the different color similarity components composed of the color information of the first color component and the color information of the third color component;
As the similarity between the same colors,
(1) The same-color similarity component consisting only of color information of the first color component;
(2) The same-color similarity component consisting only of color information of the second color component;
(3) An interpolation processing device characterized by calculating a similarity including at least one of the same-color similarity components composed only of color information of the third color component. The interpolation processing device according to claim 11,
The main similarity determination means includes
Using information about color as a feature of the image in the local area to which the subpixel to be interpolated belongs, and determining whether the image in the local area is an achromatic part or a chromatic part based on the information about the color When the image in the local region is an achromatic part, the similarity between the different colors is selected as the similarity in at least two directions in the sub-pixel to be interpolated, or the similarity between the different colors and the The weight ratio of the similarity between different colors when weighted addition of the similarity between the same colors is made larger than the weight ratio of the similarity between the same colors, and if the image in the local region is a chromatic part, interpolation is performed. The similarity between the same colors is selected as the similarity in at least two directions in the target sub-pixel, or the weight difference addition between the different color similarity and the same color similarity is performed. Interpolation processing apparatus, wherein a weighting ratio of colors similarity larger than the weighted ratio of the different color similarity between. The interpolation treatment device according to claim 14,
The main similarity determination means includes
The interpolating apparatus using the similarity between different colors as information on the color. The interpolation processing device according to claim 14 or 15,
The main similarity determination means includes
When the similarity between the different colors shows strong similarity in at least one direction, it is determined that the image in the local region is an achromatic portion, and in other cases, the image in the local region is colored. An interpolation processing device characterized in that it is determined to be a part. The interpolation processing device according to any one of claims 8 to 16,
The main similarity determination means includes
Interpolation characterized by using the similarity in at least two directions calculated not only for the interpolation target subpixel but also for the subpixels around the interpolation target subpixel as the similarity in at least two directions in the interpolation target subpixel. Processing equipment. The interpolation processing device according to claim 8 or 11,
The main similarity determination means includes
An interpolation processing apparatus, characterized in that, when the difference in similarity between directions is smaller than a predetermined threshold, the similarity in each direction is determined to be approximately the same. In the interpolation processing device according to any one of claims 2 to 7,
The fine part similarity determination means includes:
The similarity in at least two directions in the interpolation target subpixel is calculated using color information of one or more color components, and based on the similarity, the similarity of the interpolation target subpixel in at least two directions is determined. An interpolation processing device characterized by determining. 20. The interpolation processing device according to claim 19, wherein any one of the first to nth (n ≧ 2) color components is associated with one pixel, and the spatial frequency is lower than that of the first color component. When supplementing the interpolated value of the first color component to the sub-pixel constituting the pixel associated with the two color components,
The fine part similarity determination means includes:
(1) Among the subpixels that output the color information of the second color component, a similarity component indicating a difference in the color information of the second color component between the subpixels located at the shortest distance;
(2) Of the subpixels that output the color information of the first color component, a similarity component indicating a difference in color information of the first color component between the subpixels located at the shortest distance;
(3) Of the subpixel that outputs the color information of the first color component and the subpixel that outputs the color information of the second color component, the first color component and the second color component of the subpixel located at the shortest distance; An interpolation processing device characterized by calculating similarity in at least two directions in a sub-pixel to be interpolated using at least one similarity component indicating a difference in color information. The interpolation processing device according to claim 20,
The fine part similarity determination means includes:
The interpolation processing apparatus, wherein the similarity components are weighted and added in each direction to calculate the similarity in at least two directions in the interpolation target subpixel. The interpolation processing device according to claim 20 or claim 21,
The fine part similarity determination means includes:
An interpolation processing device, wherein when calculating the similarity in each direction in the interpolation target sub-pixel, each of the similarity components is normalized according to the distance between the sub-pixels. The interpolation processing device according to any one of claims 2 to 7, wherein the image data is represented by a color system composed of a plurality of color components.
The fine part similarity determination means includes:
Based on a plurality of color information selected from the color information of the interpolation target sub-pixel and the color information existing in the vicinity thereof,
(1) The similarity between different colors calculated using color information of different color components;
(2) Calculate two types of similarity with the same-color similarity calculated using the color information of the same color component, and either one according to the feature of the image in the local region to which the interpolation target subpixel belongs By switching to one of the similarities, or by switching the weighting ratio when each similarity is weighted and added, the similarity for at least two directions in the subpixel to be interpolated is calculated, and based on the similarity, at least for one direction An interpolation processing device characterized by determining the strength of similarity of interpolation target sub-pixels. The interpolation processing device according to claim 23,
The fine part similarity determination means includes:
An interpolation processing apparatus, wherein the similarity between different colors is calculated using color information existing at a distance interval shorter than the similarity between the same colors. The interpolation processing device according to any one of claims 19 to 24,
The fine part similarity determination means includes:
Interpolation characterized by using the similarity in at least two directions calculated not only for the interpolation target subpixel but also for the subpixels around the interpolation target subpixel as the similarity in at least two directions in the interpolation target subpixel. Processing equipment. The interpolation processing device according to claim 19 or claim 23,
The fine part similarity determination means includes:
An interpolation processing apparatus, characterized in that, when the difference in similarity between directions is smaller than a predetermined threshold, the similarity in each direction is determined to be approximately the same. One color component is associated with each pixel of a plurality of two-dimensionally arranged pixels, and image data generated by an imaging device in which each pixel is divided into two subpixels is assigned to each subpixel. In a recording medium recording an interpolation processing program for realizing an interpolation processing by a computer to determine an interpolation value corresponding to the color information of the missing color component,
The value of the color information output from the two subpixels constituting each pixel is used as pixel output color information corresponding to the color information of the pixel, and the similarity of the subpixels to be interpolated in at least two directions Is determined using the pixel output color information, a main similarity determination procedure;
According to the strength of the similarity determined by the main similarity determination procedure, color information of the subpixel used when calculating the interpolation value of the interpolation target subpixel is selected, and the interpolation target subpixel is used using the color information. Interpolation value calculation procedure to calculate the interpolation value of
The recording medium which recorded the interpolation processing program characterized by making a computer perform. One color component is associated with each pixel of a plurality of two-dimensionally arranged pixels, and image data generated by an imaging device in which each pixel is divided into two subpixels is assigned to each subpixel. In a recording medium recording an interpolation processing program for realizing an interpolation processing by a computer to determine an interpolation value corresponding to the color information of the missing color component,
A main similarity determination procedure for determining the strength of the similarity of the interpolation target subpixel in at least two directions using color information located at a symmetrical distance with respect to the interpolation target subpixel;
A fine part similarity determination procedure for determining the strength of the similarity of the interpolation target subpixel in at least two directions using color information located at an asymmetric distance with respect to the interpolation target subpixel;
According to the similarity determined by the main similarity determination procedure and the fine part similarity determination procedure, color information of the subpixel used when calculating the interpolation value of the interpolation target subpixel is selected, A recording medium on which an interpolation processing program is recorded, which causes a computer to execute an interpolation value calculation procedure for calculating an interpolation value of a sub-pixel to be interpolated using color information .

Images