JP3897337B2 - Road shoulder detection apparatus and method - Google Patents

Description

【００２８】
という関係式が成り立つ。ただし、
【００２９】
【数２】

【００３０】
で、h、t、g、sはカメラ１、２の３次元位置と姿勢、及びカメラ１、２それぞれに装着したレンズの焦点距離、画像中心に関するパラメーターであり、車両の中心を原点として、車両に対して相対的に停止している３次元直交座標（車両座標）系におけるパラメータである。
【００３１】
数１からXとYを消去すると、
【００３２】
【数３】

【００３３】
となる。ここで、
【００３４】
【数４】

【００３５】
は数１から数３へ式変形した際のu₁、v₁の係数と定数項であり、h、gで表される定数である。
【００３６】
数４のパラメータＨはあらかじめ固定値で与えられるものではないが、数３の関係式により、画像Ｉ１中の点(u₁,v₁)を道路平面拘束仮定した時のカメラ２から見た場合の点(u₂,v₂)の座標を求めることが可能であるから、あらかじめ与えた画像Ｉ１と画像Ｉ２の対応点の座標を数３の関係式に代入して得られるパラメータＨの各成分についての連立方程式を解いて求めることが可能である。本装置では、制御部９が数４のパラメータＨを制御部９の初期化動作の際に求める。
【００３７】
差異計算部５は数３の関係式を用いて、差異計算部６は数３を(u₁,v₁)について解いた関係式を用いて変換画像Ｍ１、Ｍ２を生成する。尚、変換画像の生成速度を向上させるため、差異計算部５、６ともに制御部９からパラメータＨを受け取った際に、数３の関係式（差異計算部６は数３を(u₁,v₁)について解いたもの）を用いて変換画像を生成するための変換テーブルＴ_1to2、Ｔ_2to1を生成して差異計算部５及び６の内部で記憶しておき、実際の変換画像生成はこの変換テーブルＴ_1to2、Ｔ_2to1を用いて行う。変換テーブルＴ_1to2、Ｔ_2to1は２次元の配列データで、Ｔ_1to2(u₁,v₁)=(u₂,v₂)、Ｔ_2to1(u₂,v₂)=(u₁,v₁)である。
【００３８】
画像Ｉnと変換画像Ｍmから求める差異画像Ｓn（n=１,m=２あるいはn=２,m=１）の各画素の輝度値は、画像Ｉnと変換画像のＭmの各画素の輝度値を数５の式で変換した値を持つ。
【００３９】
【数５】

【００４０】
ただし、Ｓn(x,y)、Ｉn(x,y)、Ｍm(x,y)は、各画像上の点(x,y)の輝度値を表す。
【００４１】
画像Ｉ１を道路平面拘束に従ってカメラ２からの見え方に変換するためのパラメータＨ_1to2を用いてカメラ２からの見え方に変換した変換画像Ｍ１では、道路に存在するガードレールや生垣などの立体物が右側に倒れこんだ形に変換される。立体物は道路と接する地点を中心にして倒れこんで変換されるので、道路より高い場所にあるもの（画素）ほど右側に大きく移動する。
【００４２】
従って、変換画像とＭ１と画像Ｉ２から差異画像Ｓ１を生成すると、道路面は輝度差が小さくなり、道路より高い場所にある物体は輝度差が大きくなる。このため、差異画像Ｓ１は図５に示すようにガードレール部分に輝度差の大きい領域Ｘ₁が現れるので、この領域Ｘ₁の境界を検出すれば路肩の位置を検出することができる。
【００４３】
画像Ｉ２を道路平面拘束に従ってカメラ１からの見え方に変換するためのパラメータＨ_2to1を用いてカメラ１からの見え方に変換した変換画像Ｍ２では、道路に存在するガードレールや生垣などの立体物が左側に倒れこんだ形に変換される。立体物は道路と接する地点を中心にして倒れこんで変換されるので、道路より高い場所にあるもの（画素）ほど左側に大きく移動する。
【００４４】
従って、変換画像とＭ２と画像Ｉ１から差異画像Ｓ２を生成すると、道路面は輝度差が小さくなり、道路より高い場所にある物体は輝度差が大きくなる。このため、差異画像Ｓ２は図５に示すようにガードレール部分に輝度差の大きい領域Ｘ₂が現れるので、この領域Ｘ₂の境界を検出すれば路肩の位置を検出することができる。
【００４５】
路肩検出部７は、差異計算部５で求めた差異画像Ｓ１から領域Ｘ₁を探索して、道路右側（カメラ２に近い側）の路肩を検出する。尚、路肩の検出には走行レーンが交わる消失点Ｖ１の座標を必要とするので、本装置では、制御部９が初期化動作の中で消失点Ｖ１の座標を求める。
【００４６】
路肩検出部７は、まず、図６（ａ）に示すように画像中の消失点Ｖ１を通る画面下端に対する水平線を求める。水平線と画像端の交点をそれぞれＨa1、Ｈa2とする。路肩検出部７は消失点Ｖ１と画像の端点Ｔm1を通る直線に沿って差異画像Ｓ１を走査する。走査範囲は端点Ｔm1が画像の端を通って反時計回りにＨa1からＨa2までである。路肩検出部７は直線Ｖ１Ｔm1を境界にして差異画像Ｓ１の画素値を比較して路肩を検出する。
【００４７】
路肩検出部７で行う画素値の比較の具体的方法を説明する。端点Ｔm1の前後所定距離（例えば5画素）離れた場所に点Ｔn1、Ｔp1をとり、三角形Ｖ１Ｔm1Ｔn1（領域Ｓn1）と三角形Ｖ１Ｔm1Ｔp1（領域Ｓp1）の領域に着目する。路肩検出部７は領域Ｓn1、Ｓp1内の平均輝度値ｍ_n1、ｍ_p1を求めて、平均輝度値と所定の閾値Ｄ₁、Ｄ₂が数６の関係を満たすかを判定する。
【００４８】
【数６】

【００４９】
路肩検出部７は、平均輝度値が数６の関係を満たした場合は、差異画像Ｓ１中の直線Ｖ１Ｔm1を路肩候補とし、数７で表される信頼度Ｒ₁を計算して路肩検出部７に内蔵されたメモリ（図示せず）に登録しておく。
【００５０】
【数７】

【００５１】
路肩検出部７は数７で計算した信頼度Ｒ₁が最小となる直線Ｖ１Ｔm1を道路右側の路肩として出力する。
【００５２】
尚、路肩検出部７では数７の式を路肩位置を判定する信頼度として用いたが、領域Ｓn1、Ｓp1の平均輝度ｍ_n1、ｍ_p1と、領域Ｓn1と領域Ｓp1を合わせた領域の平均輝度ｍ_T1を用いて、数８の関係式で定義した信頼度Ｒ₁を用いても良い。
【００５３】
【数８】

【００５４】
ただし、数８で定義した信頼度Ｒ₁を用いる場合は、信頼度Ｒ₁が最大になる直線Ｖ１Ｔm1を道路右側の路肩として出力する。
【００５５】
以上のようにして路肩検出部７は道路右側の路肩を検出し、検出した路肩の情報を制御部９へ出力する。
【００５６】
路肩検出部８は、差異計算部６で求めた差異画像Ｓ２から領域Ｘ₂を探索して、道路左側（カメラ１に近い側）の路肩を検出する。尚、路肩の検出には走行レーンが交わる消失点Ｖ２の座標を必要とするので、本装置では、制御部９が初期化動作の中で消失点Ｖ２の座標を求める。
【００５７】
路肩検出部８は、まず、図６（ｂ）に示すように画像中の消失点Ｖ２を通る画面下端に対する水平線を求める。水平線と画像端の交点をそれぞれＨb1、Ｈb2とする。路肩検出部８は消失点Ｖ２と画像の端点Ｔm2を通る直線に沿って差異画像Ｓ２を走査する。走査範囲は端点Ｔm2が画像の端を通って反時計回りにＨb1からＨb2までである。路肩検出部８は直線Ｖ２Ｔm2を境界にして差異画像Ｓ２の画素値を比較して路肩を検出する。
【００５８】
路肩検出部８で行う画素値の比較の具体的方法を説明する。端点Ｔm2の前後所定距離（例えば5画素）離れた場所に点Ｔn2、Ｔp2をとり、三角形Ｖ２Ｔm2Ｔn2（領域Ｓn2）と三角形Ｖ２Ｔm2Ｔp2（領域Ｓp2）の領域に着目する。路肩検出部８は領域Ｓn2、Ｓp2内の平均輝度値ｍ_n2、ｍ_p2を求めて、平均輝度値と所定の閾値Ｄ₁、Ｄ₂が数９の関係を満たすかを判定する。
【００５９】
【数９】

【００６０】
路肩検出部８は、平均輝度値が数９の関係を満たした場合は、差異画像Ｓ２中の直線Ｖ２Ｔm2を路肩候補とし、数１０で表される信頼度Ｒ₂を計算して路肩検出部８に内蔵されたメモリ（図示せず）に登録しておく。
【００６１】
【数１０】

【００６２】
路肩検出部８は数１０で計算した信頼度Ｒ₂が最小となる直線Ｖ２Ｔm2を道路左側の路肩として出力する。
【００６３】
尚、路肩検出部８では数１０の式を路肩位置を判定する信頼度として用いたが、領域Ｓn2、Ｓp2の平均輝度ｍ_n2、ｍ_p2と、領域Ｓn2と領域Ｓp2を合わせた領域の平均輝度ｍ_T2を用いて、数１１の関係式で定義した信頼度Ｒ₂を用いても良い。
【００６４】
【数１１】

【００６５】
ただし、数１１で定義した信頼度Ｒ₂を用いる場合は、信頼度Ｒ₂が最大になる直線Ｖ２Ｔm2を道路左側の路肩として出力する。
【００６６】
以上のようにして路肩検出部８は道路左側の路肩を検出し、検出した路肩の情報を制御部９へ出力する。
【００６７】
制御部９は前述の通り、路肩検出部７、８で検出した路肩の情報を画像情報に変換して、ＧＵＩ及び画像蓄積部３、４から読み出したカメラ１、２で撮影した画像上に合成し、これをユーザーインターフェースであるＧＵＩと合成してＧＵＩを表示部１０に出力する。この他に、本装置の使用者からの操作を受け付けたり、本装置起動時及び使用者からの要求による初期化動作を行う。
【００６８】
以下、制御部９で行う初期化動作について説明する。
【００６９】
制御部９は初期化動作において、道路平面拘束を用いて画像を変換するパラメータＨ_1to2とＨ_2to1と、走行レーンの消失点を求める。
【００７０】
制御部９はＧＵＩ上で本装置の使用者にポインティングデバイスで画像Ｉ１の道路面上の特徴点（道路平面上に描かれた直線の交点やペイントの角点等）をＮ個（４個以上）と指定したＮ個の点に対応する画像Ｉ２上の点（対応点）と、画像Ｉ１、Ｉ２それぞれの走行レーンの消失点Ｖ２、Ｖ１を指定するように促し、画像Ｉ１での道路面上の特徴点Ｎ個の座標と、それぞれの特徴点について画像Ｉ２上での対応点Ｎ個の座標と、消失点の座標Ｖ１とＶ２を得る。制御部９は消失点の座標Ｖ１を路肩検出部７に、消失点の座標Ｖ２を路肩検出部８にそれぞれ出力する。
【００７１】
制御部９は、本装置の使用者が指定したＮ組の点の座標値を、画像Ｉ１上の点を(u₁,v₁)として、画像Ｉ２上の点を(u₂,v₂)として数３の関係式に代入して得られるＨについての２Ｎ本の連立方程式を解いて、画像Ｉ１を道路平面拘束に従ってカメラ２からの見え方に変換するためのパラメータＨ_1to2を求める。制御部９は、求めたＨ_1to2を差異計算部５に出力する。
【００７２】
同様にして、制御部９は、画像Ｉ２上の点を(u₁,v₁)として、画像Ｉ１上の点を(u₂,v₂)として数３の関係式に代入して得られるＨについての２Ｎ本の連立方程式を解いて、画像Ｉ２を道路平面拘束に従ってカメラ１からの見え方に変換するためのパラメータＨ_2to1を求める。制御部９は、求めたＨ_2to1を差異計算部６に出力する。
【００７３】
以上のようにして、制御部９は初期化動作を実施する際に路肩検出処理で必要となる各種のパラメータを計算する。
【００７４】
本実施形態では差異計算部５、６で、輝度値の絶対値差分を用いて差異画像Ｓ１、Ｓ２を求めているが、これに限らず、フーリエ変換値等のテクスチャーを示す尺度や各種の相関係数を用いても構わない。
【００７５】
また、本実施形態では車両の進行方向に対して左右の路肩を検出しているが、必ずしも両側の路肩を検出する必要はない。例えば日本のように左側通行の国であれば左側の路肩のみを本装置で検出して、走行車線右側の境界は本発明以外の別の方法で白線を検出するという使い方もある。このように恒常的に左側だけを検出する場合は本実施形態の差異計算部５及び路肩検出部７の構成を省いても良い。また、逆に右側だけを検出する場合は差異計算部６及び路肩検出部８の構成を省いても良い。
【００７６】
以上、本実施形態の路肩検出装置によれば、２台のカメラで撮影したステレオ画像から、車両が走行する道路面と高さが異なる領域を抽出し、その領域と道路面との境界線を求めているので、高さや距離を求める装置に比べて少ない計算量で路肩を検出することができる。
【００７７】
（第２の実施形態）以下、本発明の第２の実施形態の路肩検出装置について説明する。本発明第２の実施形態の路肩検出装置は、第１の実施形態の路肩検出装置と大部分の構成は同じである。よって、構成の異なる部分についてだけ説明する。
【００７８】
図２は本発明第２の実施形態の路肩検出装置の路肩検出部７、８の構成を説明するブロック図である。第１の実施形態と異なり、路肩検出部７は、逆投影部１１と境界検出部１３を有し、路肩検出部８は、逆投影部１２と境界検出部１４を有している。
【００７９】
路肩検出部７は差異計算部５から得られた差異画像Ｓ１を車両が存在する道路面上に逆投影する逆投影部１１へ入力して、逆投影部１１からの逆投影した差異画像ＲＳ１から、道路面と高さが異なる領域を求め、その境界線を道路の右側の路肩として検出する境界検出部１３を用いて路肩を検出している。
【００８０】
路肩検出部８は差異計算部６から得られた差異画像Ｓ２を車両が存在する道路面上に逆投影する逆投影部１２へ入力して、逆投影部１２からの逆投影した差異画像ＲＳ２から、道路面と高さが異なる領域を求め、その境界線を道路の右側の路肩として検出する境界検出部１４を用いて路肩を検出している。
【００８１】
逆投影部１１は、差異計算部５で求めた差異画像Ｓ１を数１２の式により、車両が存在する道路面上に逆投影して逆投影差異画像ＲＳ１を生成する。図７（ａ）に示すような態様をしている差異画像Ｓ１は、車両が存在する道路面上に逆投影されると図７（ｂ）に示すような逆投影差異画像ＲＳ１になる。逆投影部１１は生成した逆投影差異画像ＲＳ１を境界検出部１３へ出力する。
【００８２】
【数１２】

【００８３】
数１２において、(x,y)は図７（ａ）に示すように差異画像Ｓ１中の点であり、座標系の原点は差異画像Ｓ１の消失点位置である。また、(u,v)は図７（ｂ）に示すように逆投影画像中の点であり、座標系の原点は逆投影差異画像ＲＳ１の上部中央位置である。y_minは差異画像Ｓ１上において逆投影変換を行う下限ラインであり、Ｒ_x、Ｒ_yは逆投影差異画像ＲＳ１を適正な画像サイズに収めるための縮小率である。y_min、Ｒ_x、Ｒ_yは制御部９が初期化動作を行う時に、本装置の使用者にそれぞれの値を設定するように促して設定される値である。
【００８４】
逆投影画像では、ガードレール、壁面、生垣、歩道の段差などの路肩部分が道路と路肩の境界位置から外側に倒れこむ形で表現される。また、道路部分は道路を真上から見下ろす形で表現されるので、逆投影前の画像では消失点に向かって幅が狭くなっていく一定幅の直線道路が、逆投影画像では一定幅の道路として表現される。
【００８５】
境界検出部１３は、逆投影画像ＲＳ１を走査して道路と道路右側路肩との境界を検出する。境界検出部１３は、逆投影画像ＲＳ１を図９（ｂ）に示すように水平方向の領域Ｗ_i2(i=1〜n)に分割して、各領域Ｗ_i2内を水平方向に走査する。境界検出部１３は図９（ｂ）に示すように、走査地点Ｐ２を中心に、高さが領域Ｗ_i2と同じ高さで、予め設定しておいた固定幅ｄ_l2、ｄ_r2を有する領域Ｑ_l2、Ｑ_r2を設定して、領域Ｑ_l2、Ｑ_r2内の平均輝度値ｍ_Ql2、ｍ_Qr2をそれぞれ計算して、所定の閾値ＤＤ₁、ＤＤ₂と数１３の関係を満たすかを判定する。
【００８６】
【数１３】

【００８７】
境界検出部１３は、平均輝度値が数１３の関係を満たした場合は、逆投影画像ＲＳ１中の走査地点Ｐ２を領域Ｗ_i2内の路肩境界線候補地点とし、数１４で表される信頼度ＲＲ_P2を計算して境界検出部１３に内蔵のメモリ（図示せず）に登録しておく。
【００８８】
【数１４】

【００８９】
境界検出部１３は、各領域Ｗ_i2毎に、信頼度ＲＲ_P2の値が最小となる走査地点Ｐ２を領域Ｗ_i2内に存在する道路右側の路肩位置として、座標のu成分を走査地点Ｐ２の水平位置、v成分を領域Ｗ_i2の高さ方向の中点のv成分とする路肩位置代表点Ａ_i2を境界検出部１３に内蔵のメモリ（図示せず）に順次記憶する。境界検出部１３は、図１０（ｂ）に示すように順次記憶した路肩位置代表点Ａ_i2を結んだ折れ線を道路右側の路肩情報として制御部９へ出力する。
【００９０】
尚、境界検出部１３では路肩位置を判定する信頼度として数１４の関係式を用いたが、領域Ｑ_l2とＱ_r2を合わせた領域の平均輝度値ｍ_T2を求めて数１５の関係式を信頼度として用いても良い。
【００９１】
【数１５】

【００９２】
数１５を信頼度ＲＲ_P2として用いた場合は、信頼度ＲＲ_P2が最大となる走査地点Ｐ２を領域Ｗ_i2の路肩位置とみなせばよい。
【００９３】
逆投影部１２は、差異計算部５で求めた差異画像Ｓ２を数１２の式により、車両が存在する道路面上に逆投影して図８（ａ）に示すような逆投影差異画像ＲＳ２を生成する。逆投影部１２は生成した逆投影差異画像ＲＳ２を境界検出部１４へ出力する。
【００９４】
境界検出部１４は、逆投影画像ＲＳ２を走査して道路と道路左側路肩との境界を検出する。境界検出部１４は、逆投影画像ＲＳ２を図９（ａ）に示すように水平方向の領域Ｗ_i1(i=1〜n)に分割して、各領域Ｗ_i1内を水平方向に走査する。境界検出部１４は図９（ａ）に示すように、走査地点Ｐ１を中心に、高さが領域Ｗ_i1と同じ高さで、予め設定しておいた固定幅ｄ_l1、ｄ_r1を有する領域Ｑ_l1、Ｑ_r1を設定して、領域Ｑ_l1、Ｑ_r1内の平均輝度値ｍ_Ql1、ｍ_Qr1をそれぞれ計算して、所定の閾値ＤＤ₁、ＤＤ₂と数１６の関係を満たすかを判定する。
【００９５】
【数１６】

【００９６】
境界検出部１４は、平均輝度値が数１６の関係を満たした場合は、逆投影画像ＲＳ２中の走査地点Ｐ１を領域Ｗ_i1内の路肩境界線候補地点とし、数１７で表される信頼度ＲＲ_P1を計算して境界検出部１４に内蔵のメモリ（図示せず）に登録しておく。
【００９７】
【数１７】

【００９８】
境界検出部１４は、各領域Ｗ_i1毎に、信頼度ＲＲ_P1の値が最小となる走査地点Ｐ１を領域Ｗ_i1内に存在する道路右側の路肩位置として、座標のu成分を走査地点Ｐ１の水平位置、v成分を領域Ｗ_i1の高さ方向の中点のv成分とする路肩位置代表点Ａ_i1を境界検出部１４に内蔵のメモリ（図示せず）に順次記憶する。境界検出部１４は、図１０（ａ）に示すように順次記憶した路肩位置代表点Ａ_i1を結んだ折れ線を道路左側の路肩情報として制御部９へ出力する。
【００９９】
尚、境界検出部１３では路肩位置を判定する信頼度として数１７の関係式を用いたが、領域Ｑ_l1とＱ_r1を合わせた領域の平均輝度値ｍ_T1を求めて数１８の関係式を信頼度として用いても良い。
【０１００】
【数１８】

【０１０１】
数１８を信頼度ＲＲ_P1として用いた場合は、信頼度ＲＲ_P1が最大となる走査地点Ｐ１を領域Ｗ_i1の路肩位置とみなせばよい。
【０１０２】
本実施形態においても第１の実施形態と同様で、車両の進行方向に対して片側の路肩を検出すればよい場合は、検出を行わない側の差異計算部及び路肩検出部の構成を省略しても構わない。
【０１０３】
以上、本実施形態の路肩検出装置によれば、第１の実施形態の路肩検出装置の利点に加えて、路肩検出の際の画像の走査方向が画面に対して水平方向であるので、特にハードウエアで実装する場合に実装が容易であるという利点がある。
【０１０４】
（第３の実施形態）図１１は本発明の路肩検出装置を用いた自動車の車両制御装置の構成を説明するブロック図である。
【０１０５】
本実施形態の車両制御装置の路肩検出装置部分の構成は、第１及び第２の実施形態で説明した路肩検出装置とほぼ同等である。異なる点は、初期化動作を行う制御部９と検出結果を表示するための表示部１０を取り除いた点である。尚、路肩検出装置部分としては、第１の実施形態の路肩検出装置を用いても、第２の実施形態の路肩検出装置のいずれを用いても構わない。
【０１０６】
出力部１５は路肩の検出結果を路肩検出装置部分の外部に出力するためのインターフェースである。
【０１０７】
車両制御部１６は出力部１５から出力された路肩検出結果をもとに、本装置を設置した車両の走行車線における位置を判断して、走行車線から外れてそうな場合に運転者に注意を促したり、事故や衝突の危険がある場合にはステアリングやブレーキを操作して回避するための制御部分である。
【０１０８】
路肩検出装置部分での路肩検出処理に必要なＨ_1to2、Ｈ_2to1、走行レーン消失点などの各種パラメータは、予めパラメータ算出用装置（図示せず）を用いて算出して、ＲＯＭ（Ｒｅａｄ Ｏｎｌｙ Ｍｅｍｏｒｙ）に記憶させて工場出荷時に組み込んで与える。そのため、第１、第２の実施形態のような初期化動作を行う機能は不要となっている。
【０１０９】
以上、本実施形態の車両制御装置ならば、路肩を自動的に検出して路肩から外れそうになったときは注意を促し、場合によっては回避動作を自動的に行うことが可能なので、居眠り運転などによる事故を減らせる。
【０１１０】
【発明の効果】
本発明によれば、２台のカメラで撮影したステレオ画像から車両が走行する道路面と高さが異なる領域を抽出し、その領域と道路面との境界線を求めることで安定かつ計算量の少ない路肩検出装置及びその方法を提供することができる。
【図面の簡単な説明】
【図１】 本発明の第１、第２の実施形態の路肩検出装置の構成を示すブロック図。
【図２】 本発明の第２の実施形態の路肩検出装置の路肩検出部の構成を説明するブロック図。
【図３】 本発明の第１、第２の実施形態の路肩検出装置におけるカメラの設置位置を示す図。
【図４】 （ａ）カメラ１から得られる画像Ｉ１を説明する図。（ｂ）カメラ２から得られる画像Ｉ２を説明する図。
【図５】 （ａ）差異画像Ｓ２を説明する図。（ｂ）差異画像Ｓ１を説明する図。
【図６】 （ａ）本発明の第１の実施形態の路肩検出装置で差異画像Ｓ２から路肩を検出する方法を説明する図。（ｂ）本発明の第１の実施形態の路肩検出装置で差異画像Ｓ１から路肩を検出する方法を説明する図。
【図７】 （ａ）カメラで得られる画像を説明する図。（ｂ）逆投影画像を説明する図。
【図８】 （ａ）差異画像Ｓ２の逆投影画像ＲＳ２を説明する図。（ｂ）差異画像Ｓ１の逆投影画像ＲＳ１を説明する図。
【図９】 （ａ）本発明の第２の実施形態の路肩検出装置で、逆投影画像ＲＳ２から路肩を検出する方法を説明する図。（ｂ）本発明の第２の実施形態の路肩検出装置で、逆投影画像ＲＳ１から路肩を検出する方法を説明する図。
【図１０】 （ａ）本発明の第２の実施形態の路肩検出装置で検出されたカメラ１側の路肩を示す折れ線を説明する（ｂ）本発明の第２の実施形態の路肩検出装置で検出されたカメラ２側の路肩を示す折れ線を説明する図。
【図１１】 本発明の第３の実施形態である、車両制御装置の構成を説明する図。
【符号の説明】
１、２ カメラ
３、４ 画像蓄積部
５、６ 差異計算部
７、８ 路肩検出部
９ 制御部
１０ 表示部
１１、１２ 逆投影部
１３、１４ 境界検出部
Ｖ１、Ｖ２ 消失点
Ｐ１、Ｐ２ 走査地点[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a road shoulder detection apparatus and method for detecting a road shoulder from an image taken by a camera mounted on an automobile.
[0002]
[Prior art]
In an alarm system for ensuring safe driving of an automobile and an automatic driving system of an automobile, a technique for detecting whether or not a running vehicle departs from a driving lane is required. In order to detect whether the vehicle deviates from the traveling lane, it is necessary to detect both left and right ends of the traveling lane. When detecting both ends of a driving lane using an image taken by an in-vehicle camera, a method of detecting a lane marker such as a white line on the road surface is common. However, this method can be applied only to roads having lane markers on the road surface. The presence of lane markers such as white lines can be expected on expressways and wide general roads that have been maintained, but there is no center line or white line indicating the end of the road on a relatively narrow general road. There are many cases. In such a road where the driving lane is not clearly indicated by a white line or the like, it is necessary to detect sidewalks, median strips, side walls, guardrails, hedges, and the like, and detect them as road shoulders.
[0003]
2. Description of the Related Art Conventionally, as a road shoulder detection method, a method for obtaining a road shoulder using information such as image brightness, color, or texture has been proposed. For example, CEThorpe, MHHerbert and T. Kanade, "Vision and Navigation for the Carnegie-Mellone Navlab," IEEE Trans. PAMI, vol.01, no.3, pp.362-373, 1988. The area was extracted as a road area, and both left and right edges were detected as road shoulders. In LSDavis, D. Dementhon, S. Dickinson and P. Veatch, "Algorithms for road navigation," Vision-based vehicle Guidance, Springer-Verlag, Chapter 3, 1992. Those having high strength and having long edge segments were detected as road shoulders.
[0004]
However, there are many pixels and regions in the image that have similar brightness, color, or texture to the road. In addition, the brightness level, color, and texture of the road surface itself can change from moment to moment due to changes in weather, seasons, and time, even with images taken from the same location, so the road shoulder can be detected stably. Have difficulty.
[0005]
In addition, many areas with long edge segments appear in the image, such as the boundaries of buildings and utility pole shadows on the road, and these shadows change from moment to moment, so the road shoulder can be detected stably and accurately. Is difficult.
[0006]
As another method, Japanese Patent Application Laid-Open No. 11-213138 proposes a method of detecting side walls such as guardrails by obtaining height and distance information from stereo images obtained by two cameras. In this method, the corresponding pixels between the stereo images are searched and the height and distance information is calculated to calculate the three-dimensional coordinate data. However, a large calculation cost is required for the pixel search and the grouping of the three-dimensional coordinate data. Is required.
[0007]
[Problems to be solved by the invention]
As described above, in the conventional method using information such as the brightness, color, texture, etc. of the photographed image, a side wall having a brightness, color, or texture similar to that of a road is erroneously detected as a road, and the shoulder position is obtained correctly. There is a problem of not.
[0008]
Further, the method using the edge segment has a problem that a long edge segment such as a boundary of a shadow projected on the road is erroneously detected as a road shoulder.
[0009]
Furthermore, the method of calculating the height and distance information from the stereo images obtained by the two cameras and detecting the side wall has a problem that the calculation amount is large.
[0010]
The present invention has been made in view of the above problems, and an object of the present invention is to provide a road shoulder detection apparatus and method that are stable and have a small amount of calculation using stereo images taken by two cameras.
[0011]
[Means for Solving the Problems]
In order to solve the above problems, a shoulder detection method according to the present invention is a method for detecting a shoulder of a road from a traveling vehicle, wherein the first shoulder shoulder is installed in a direction perpendicular to the traveling direction of the vehicle. An image acquisition step of acquiring first and second images simultaneously using a second camera; an image storage step of storing the first image and the second image in a memory; When it is assumed that an arbitrary pixel exists on the road surface on which the vehicle travels, the correspondence between the first image and the view from the second camera is used, and the first image and the first image The second image when the difference between the second images is calculated to generate the first difference image, and it is assumed that any pixel of the second image exists on the road surface on which the vehicle travels To convert the image to the view from the first camera A difference image generating step of calculating a difference between the first image and the second image to generate a first difference image; and a position where the second image is acquired from the first difference image. It has a road shoulder detection step for detecting a road shoulder on the near side and detecting a road shoulder on the side close to the position where the first image is acquired from the second difference image.
[0012]
The road shoulder detection device of the present invention is a device that is installed in a vehicle traveling on a road and detects a road shoulder of the road, and is installed in the vehicle with a predetermined interval in a direction perpendicular to the traveling direction of the vehicle. The first and second cameras that simultaneously capture two images in front of the vehicle, the first image storage unit that stores the first image captured by the first camera in screen units, and the second camera The second image storage unit that stores the captured second image in units of screens, and the first image when it is assumed that an arbitrary pixel of the first image exists on the road surface on which the vehicle travels Difference between the first image from the first image accumulating unit and the second image from the second image accumulating unit using a correspondence relationship for converting the image into a view from the second camera A first difference image generation unit for calculating a first difference image and calculating the second difference image When it is assumed that an arbitrary pixel of the image exists on a road surface on which the vehicle travels, the first image is used by using a correspondence relationship for converting the second image into a view from the first camera. A second difference image generation unit that calculates a difference between the first image from the storage unit and the second image from the second image storage unit to generate a second difference image; and A first shoulder detection unit that detects a shoulder on the side closer to the second camera from the difference image, and a first shoulder detection unit that detects a shoulder closer to the second camera from the first difference image; It is characterized by having.
[0013]
As described above, according to the present invention, a road shoulder can be detected stably with fewer false detections compared to a road shoulder detection method and apparatus using an edge segment and the like, and a conventional road shoulder detection method and apparatus using a stereo image can be used. Since no height or distance information is calculated, it is possible to detect a road shoulder with a small amount of calculation.
[0014]
DETAILED DESCRIPTION OF THE INVENTION
(First Embodiment) The first embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram illustrating a configuration of a road shoulder detection device according to a first embodiment of the present invention.
[0015]
The road shoulder detection apparatus of this embodiment includes cameras 1 and 2 for capturing moving images, and image storage units 3 and 4 that temporarily store the captured moving images sequentially for each image. The apparatus further includes difference calculation units 5 and 6 for obtaining difference images S1 and S2 from images taken by the cameras 1 and 2, and road shoulder detection units 7 and 8 for detecting a road shoulder using the difference images. The apparatus further provides a GUI as an interface for the user of the apparatus, controls the operation of each part of the apparatus, and images and images detected by the GUI and cameras 1 and 2 described above. And a display unit 10 for displaying a road shoulder.
[0016]
This device is for mounting on a car. As shown in FIG. 3, the cameras 1 and 2 are provided with a camera 1 on the left side and a camera 2 on the right side in the traveling direction of the vehicle. Specifically, the camera 1 is installed at the inner left end of the front window of the car, and the camera 2 is installed at the inner right end. The distance between the cameras and the height of the camera installation location from the road surface depends on the size and type of the vehicle, but in the case of a passenger car, the distance between the cameras is about 1.5m to 2.0m and the height is about 70cm to 1.5m. Install. Then, the orientation is adjusted so that the visual fields of the camera 1 and the camera 2 overlap to some extent at the time of installation.
[0017]
The cameras 1 and 2 are visible light region cameras using a CCD (charge coupled device) as a light receiving portion. FIG. 4 is a diagram for explaining images taken by the cameras 1 and 2. FIG. 4A shows a case where the camera 1 on the left side of the vehicle is taken, and FIG. It is how it looks. An image I1 captured by the camera 1 is output to the image storage unit 3, and an image I2 captured by the camera 2 is output to the image storage unit 4.
[0018]
The image storage unit 3 temporarily stores an image from the camera 1 in units of screens, and the image storage unit 4 temporarily stores an image from the camera 2 in units of screens.
[0019]
The difference calculation unit 5 converts the image I1 into the converted image M1, and then generates a difference image S1 from the converted image M1 and the image I2. The difference calculation unit 5 first converts the image I1 into a view from the camera 2 when it is assumed that all pixels in the image I1 are present on the road plane on which the vehicle is traveling (road plane constraint assumption). Thus, the converted image M1 is generated. Then, the difference image S1 is generated by calculating the absolute value difference in luminance between the converted image M1 and the image I2 for each pixel.
[0020]
Similarly, the difference calculation unit 6 converts the image I2 into the converted image M2, and then generates a difference image S2 from the converted image M2 and the image I1. The difference calculation unit 6 first converts the image I2 into a view from the camera 1 when all the pixels are assumed to be road plane constraints, and generates a converted image M2. Then, an absolute value difference between the image I1 and the converted image M2 is calculated for each pixel to generate a difference image S2.
[0021]
The road shoulder detection unit 7 detects the road shoulder on the right side (camera 2 side) of the road from the difference image S1 obtained by the difference calculation unit 5, and the road shoulder detection unit 8 calculates the road from the difference image S2 obtained by the difference calculation unit 6. The left shoulder (camera 1 side) is detected.
[0022]
The control unit 9 generates a shoulder information image representing the information on the shoulder detected by the shoulder detection units 7 and 8 and synthesizes it on the images I1 and I2 taken by the cameras 1 and 2 read from the image storage units 3 and 4. A traveling road image is generated. The control unit 9 synthesizes the generated two traveling road images (corresponding to I1 and I2) into a frame for displaying the traveling road image provided on the GUI, and outputs the GUI to the display unit 10. To do.
[0023]
Further, the control unit 9 receives an operation from a user of the apparatus through a pointing device connected to the control unit 9. Then, the control unit 9 performs the initialization operation of the apparatus at the time of installation of the apparatus or according to a request from the user. In this initialization operation, various parameters necessary for detecting a road shoulder are calculated as described later.
[0024]
The display unit 10 displays the GUI output from the control unit 9.
[0025]
Hereinafter, a method for generating the converted images M1 and M2 and a method for generating the difference images S1 and S2 performed by the difference calculation units 5 and 6 will be described.
[0026]
Arbitrary points (X, Y) on the road plane (XY plane) are projected onto the image plane of camera 1 and the image plane of camera 2 (u ₁ , v ₁ ), (U ₂ , v ₂ ). When the road plane constraint assumption is made,
[0027]
[Expression 1]

[0028]
The following relational expression holds. However,
[0029]
[Expression 2]

[0030]
H, t, g, and s are parameters related to the three-dimensional position and orientation of the cameras 1 and 2 and the focal length of the lens mounted on each of the cameras 1 and 2 and the center of the image. Is a parameter in a three-dimensional orthogonal coordinate (vehicle coordinate) system that is relatively stopped.
[0031]
Erasing X and Y from Equation 1
[0032]
[Equation 3]

[0033]
It becomes. here,
[0034]
[Expression 4]

[0035]
Is u when the equation is transformed from Equation 1 to Equation 3. ₁ , V ₁ Are constants represented by h and g.
[0036]
Although the parameter H in Expression 4 is not given as a fixed value in advance, the point (u) in the image I1 is expressed by the relational expression in Expression 3. ₁ , v ₁ ) As seen from the camera 2 when the road plane constraint is assumed (u ₂ , v ₂ ) Can be obtained by solving the simultaneous equations for each component of the parameter H obtained by substituting the coordinates of the corresponding points of the images I1 and I2 given in advance into the relational expression (3). It is possible. In this apparatus, the control unit 9 obtains the parameter H of Equation 4 during the initialization operation of the control unit 9.
[0037]
The difference calculation unit 5 uses the relational expression of Equation 3, and the difference calculation unit 6 calculates Equation 3 (u ₁ , v ₁ The converted images M1 and M2 are generated by using the relational expression solved for). In order to improve the generation speed of the converted image, when the difference calculation units 5 and 6 both receive the parameter H from the control unit 9, the relational expression of Equation 3 (the difference calculation unit 6 ₁ , v ₁ Conversion table T for generating a conversion image using _1to2 , T _2to1 Is generated and stored in the difference calculation units 5 and 6, and the actual conversion image generation is performed in the conversion table T. _1to2 , T _2to1 To do. Conversion table T _1to2 , T _2to1 Is two-dimensional array data, T _1to2 (u ₁ , v ₁ ) = (u ₂ , v ₂ ), T _2to1 (u ₂ , v ₂ ) = (u ₁ , v ₁ ).
[0038]
The luminance value of each pixel of the difference image Sn (n = 1, m = 2 or n = 2, m = 1) obtained from the image In and the converted image Mm is the luminance value of each pixel of the image In and the converted image Mm. It has a value converted by the formula (5).
[0039]
[Equation 5]

[0040]
However, Sn (x, y), In (x, y), and Mm (x, y) represent the luminance value of the point (x, y) on each image.
[0041]
Parameter H for converting the image I1 into a view from the camera 2 in accordance with the road plane constraint _1to2 In the converted image M1 converted into the view from the camera 2 by using the, a three-dimensional object such as a guardrail or a hedge existing on the road is converted into a shape collapsed to the right side. Since the three-dimensional object falls down and is converted around a point in contact with the road, the object (pixel) located higher than the road moves to the right.
[0042]
Therefore, when the difference image S1 is generated from the converted image, M1, and the image I2, the road surface has a small luminance difference, and an object located higher than the road has a large luminance difference. Therefore, the difference image S1 is an area X having a large luminance difference in the guardrail portion as shown in FIG. ₁ Appears, so this region X ₁ The position of the road shoulder can be detected by detecting the boundary.
[0043]
Parameter H for converting the image I2 into a view from the camera 1 according to the road plane constraint _2to1 In the converted image M2 converted into a view from the camera 1 by using a three-dimensional object such as a guardrail or a hedge existing on the road, the converted image is converted into a shape in which it falls to the left side. Since the three-dimensional object falls down and is converted around a point in contact with the road, the object (pixel) located higher than the road moves to the left.
[0044]
Therefore, when the difference image S2 is generated from the converted image, M2, and the image I1, the road surface has a small luminance difference, and an object at a location higher than the road has a large luminance difference. Therefore, the difference image S2 is a region X having a large luminance difference in the guardrail portion as shown in FIG. ₂ Appears, so this region X ₂ The position of the road shoulder can be detected by detecting the boundary.
[0045]
The road shoulder detection unit 7 calculates the region X from the difference image S1 obtained by the difference calculation unit 5. ₁ To detect the shoulder on the right side of the road (side closer to the camera 2). In addition, since the detection of the road shoulder requires the coordinates of the vanishing point V1 where the traveling lanes intersect, in this apparatus, the control unit 9 obtains the coordinates of the vanishing point V1 during the initialization operation.
[0046]
The road shoulder detection unit 7 first obtains a horizontal line with respect to the lower end of the screen passing through the vanishing point V1 in the image as shown in FIG. Let the intersections of the horizontal line and the image edge be Ha1 and Ha2, respectively. The road shoulder detection unit 7 scans the difference image S1 along a straight line passing through the vanishing point V1 and the end point Tm1 of the image. The scanning range is H when the end point Tm1 passes counterclockwise through the edge of the image. a 1 to H a Up to 2. The road shoulder detection unit 7 detects the road shoulder by comparing the pixel values of the difference image S1 with the straight line V1Tm1 as a boundary.
[0047]
A specific method of comparing pixel values performed by the road shoulder detection unit 7 will be described. Points Tn1 and Tp1 are taken at predetermined distances (for example, 5 pixels) before and after the end point Tm1, and attention is paid to the regions of the triangle V1Tm1Tn1 (region Sn1) and the triangle V1Tm1Tp1 (region Sp1). The shoulder detection unit 7 calculates the average luminance value m in the areas Sn1 and Sp1. _n1 , M _p1 For the average luminance value and a predetermined threshold D ₁ , D ₂ Whether or not satisfies the relationship of Equation (6).
[0048]
[Formula 6]

[0049]
When the average brightness value satisfies the relationship of Equation 6, the road shoulder detection unit 7 sets the straight line V1Tm1 in the difference image S1 as a road shoulder candidate, and the reliability R expressed by Equation 7: ₁ And is registered in a memory (not shown) built in the road shoulder detection unit 7.
[0050]
[Expression 7]

[0051]
The road shoulder detection unit 7 has a reliability R calculated by Equation 7 ₁ The straight line V1Tm1 that minimizes is output as the shoulder on the right side of the road.
[0052]
The road shoulder detection unit 7 uses Equation 7 as the reliability for determining the road shoulder position, but the average brightness m of the regions Sn1 and Sp1. _n1 , M _p1 And the average luminance m of the region Sn1 and region Sp1 combined _T1 The reliability R defined by the relational expression of Equation 8 using ₁ May be used.
[0053]
[Equation 8]

[0054]
However, the reliability R defined in Equation 8 ₁ Is used, the reliability R ₁ A straight line V1Tm1 that maximizes is output as the shoulder on the right side of the road.
[0055]
As described above, the road shoulder detection unit 7 detects the road shoulder on the right side of the road and outputs the detected road shoulder information to the control unit 9.
[0056]
The road shoulder detection unit 8 calculates the region X from the difference image S2 obtained by the difference calculation unit 6. ₂ And the shoulder on the left side of the road (side closer to the camera 1) is detected. In addition, since the detection of the road shoulder requires the coordinates of the vanishing point V2 where the traveling lanes intersect, in this apparatus, the control unit 9 obtains the coordinates of the vanishing point V2 during the initialization operation.
[0057]
First, the road shoulder detection unit 8 obtains a horizontal line with respect to the lower end of the screen passing through the vanishing point V2 in the image as shown in FIG. Let the intersections of the horizontal line and the image edge be Hb1 and Hb2, respectively. The road shoulder detection unit 8 scans the difference image S2 along a straight line passing through the vanishing point V2 and the end point Tm2 of the image. The scanning range is from Hb1 to Hb2 with the end point Tm2 passing counterclockwise through the edge of the image. The road shoulder detection unit 8 detects the road shoulder by comparing the pixel values of the difference image S2 with the straight line V2Tm2 as a boundary.
[0058]
A specific method of comparing pixel values performed by the road shoulder detection unit 8 will be described. Points Tn2 and Tp2 are taken at predetermined distances (for example, 5 pixels) before and after the end point Tm2, and attention is paid to the regions of the triangle V2Tm2Tn2 (region Sn2) and the triangle V2Tm2Tp2 (region Sp2). The road shoulder detection unit 8 calculates the average luminance value m in the areas Sn2 and Sp2. _n2 , M _p2 For the average luminance value and a predetermined threshold D ₁ , D ₂ Determines whether the relationship of Equation 9 is satisfied.
[0059]
[Equation 9]

[0060]
When the average luminance value satisfies the relationship of Equation 9, the road shoulder detection unit 8 sets the straight line V2Tm2 in the difference image S2 as a road shoulder candidate, and the reliability R expressed by Equation 10: ₂ And is registered in a memory (not shown) built in the road shoulder detection unit 8.
[0061]
[Expression 10]

[0062]
The road shoulder detection unit 8 has the reliability R calculated by Equation 10 ₂ The straight line V2Tm2 that minimizes is output as the shoulder on the left side of the road.
[0063]
The road shoulder detection unit 8 uses the equation (10) as the reliability for determining the road shoulder position, but the average luminance m of the regions Sn2 and Sp2 _n2 , M _p2 And the average luminance m of the region Sn2 and the region Sp2 combined _T2 The reliability R defined by the relational expression of Equation 11 using ₂ May be used.
[0064]
[Expression 11]

[0065]
However, the reliability R defined in Equation 11 ₂ Is used, the reliability R ₂ A straight line V2Tm2 that maximizes is output as the shoulder on the left side of the road.
[0066]
As described above, the road shoulder detection unit 8 detects the road shoulder on the left side of the road and outputs the detected road shoulder information to the control unit 9.
[0067]
As described above, the control unit 9 converts the road shoulder information detected by the road shoulder detection units 7 and 8 into image information, and synthesizes the image information captured by the cameras 1 and 2 read from the GUI and image storage units 3 and 4. Then, this is combined with a GUI that is a user interface, and the GUI is output to the display unit 10. In addition to this, an operation from the user of the apparatus is accepted, or an initialization operation is performed when the apparatus is activated and requested by the user.
[0068]
Hereinafter, the initialization operation performed by the control unit 9 will be described.
[0069]
In the initialization operation, the control unit 9 uses a parameter H for converting an image using road plane constraints. _1to2 And H _2to1 Then, the vanishing point of the driving lane is obtained.
[0070]
The control unit 9 allows the user of the apparatus on the GUI to use N (4 or more) feature points on the road surface of the image I1 (intersections of straight lines drawn on the road plane, corner points of paint, etc.) with a pointing device. ) And the points (corresponding points) on the image I2 corresponding to the N points designated, and the vanishing points V2 and V1 of the traveling lanes of the images I1 and I2, respectively, on the road surface in the image I1. The coordinates of N feature points, the coordinates of N corresponding points on the image I2 and the coordinates V1 and V2 of vanishing points are obtained for each feature point. The control unit 9 outputs the vanishing point coordinates V1 to the road shoulder detection unit 7 and the vanishing point coordinates V2 to the road shoulder detection unit 8, respectively.
[0071]
The control unit 9 uses the coordinate values of N sets of points designated by the user of this apparatus as the points on the image I1 (u ₁ , v ₁ ) As a point on image I2 (u ₂ , v ₂ ) To solve the 2N simultaneous equations for H obtained by substituting into the relational expression of Equation 3, and to convert the image I1 into a view from the camera 2 according to the road plane constraint. _1to2 Ask for. The control unit 9 calculates the calculated H _1to2 Is output to the difference calculation unit 5.
[0072]
Similarly, the control unit 9 changes the point on the image I2 to (u ₁ , v ₁ ) As a point on image I1 (u ₂ , v ₂ ) To solve the 2N simultaneous equations for H obtained by substituting into the relational expression of Equation 3, and convert the image I2 into a view from the camera 1 according to the road plane constraint. _2to1 Ask for. The control unit 9 calculates the calculated H _2to1 Is output to the difference calculation unit 6.
[0073]
As described above, the control unit 9 calculates various parameters necessary for the road shoulder detection process when performing the initialization operation.
[0074]
In the present embodiment, the difference calculation units 5 and 6 obtain the difference images S1 and S2 using the absolute value difference of the luminance values. However, the present invention is not limited to this, and a scale indicating various textures such as a Fourier transform value and various phases. The relation number may be used.
[0075]
In the present embodiment, the left and right road shoulders are detected with respect to the traveling direction of the vehicle, but it is not always necessary to detect the road shoulders on both sides. For example, in a left-handed country such as Japan, only the left shoulder is detected by this device, and the white line is detected by another method other than the present invention at the boundary on the right side of the traveling lane. When only the left side is detected in this way, the configurations of the difference calculation unit 5 and the road shoulder detection unit 7 of this embodiment may be omitted. Conversely, when only the right side is detected, the configurations of the difference calculation unit 6 and the road shoulder detection unit 8 may be omitted.
[0076]
As described above, according to the road shoulder detection apparatus of the present embodiment, an area having a height different from the road surface on which the vehicle travels is extracted from the stereo image captured by the two cameras, and the boundary line between the area and the road surface is obtained. Therefore, the road shoulder can be detected with a small amount of calculation compared to a device that obtains the height and distance.
[0077]
(Second Embodiment) A road shoulder detection apparatus according to a second embodiment of the present invention will be described below. The road shoulder detection apparatus according to the second embodiment of the present invention has the same configuration as the road shoulder detection apparatus according to the first embodiment. Therefore, only the different parts will be described.
[0078]
FIG. 2 is a block diagram illustrating the configuration of the shoulder detection units 7 and 8 of the shoulder detection device according to the second embodiment of the present invention. Unlike the first embodiment, the road shoulder detection unit 7 includes a back projection unit 11 and a boundary detection unit 13, and the road shoulder detection unit 8 includes a back projection unit 12 and a boundary detection unit 14.
[0079]
The road shoulder detection unit 7 inputs the difference image S1 obtained from the difference calculation unit 5 to the backprojection unit 11 that backprojects the difference image S1 onto the road surface on which the vehicle exists, and from the backprojected difference image RS1 from the backprojection unit 11. An area having a height different from that of the road surface is obtained, and a road shoulder is detected using a boundary detection unit 13 that detects the boundary line as a road shoulder on the right side of the road.
[0080]
The road shoulder detection unit 8 inputs the difference image S2 obtained from the difference calculation unit 6 to the back projection unit 12 that back projects onto the road surface on which the vehicle exists, and from the back projected difference image RS2 from the back projection unit 12. An area having a height different from the road surface is obtained, and a road shoulder is detected by using a boundary detection unit 14 that detects the boundary line as a road shoulder on the right side of the road.
[0081]
The backprojection unit 11 backprojects the difference image S1 obtained by the difference calculation unit 5 on the road surface on which the vehicle is present according to the equation (12) to generate a backprojection difference image RS1. When the difference image S1 having the form shown in FIG. 7A is back-projected on the road surface on which the vehicle exists, the difference image S1 becomes a back-projection difference image RS1 as shown in FIG. 7B. The backprojection unit 11 outputs the generated backprojection difference image RS1 to the boundary detection unit 13.
[0082]
[Expression 12]

[0083]
In Equation 12, (x, y) is a point in the difference image S1 as shown in FIG. 7A, and the origin of the coordinate system is the vanishing point position of the difference image S1. Further, (u, v) is a point in the backprojection image as shown in FIG. 7B, and the origin of the coordinate system is the upper center position of the backprojection difference image RS1. y _min Is a lower limit line for performing back projection transformation on the difference image S1, and R _x , R _y Is a reduction ratio for accommodating the backprojection difference image RS1 in an appropriate image size. y _min , R _x , R _y Is a value set by prompting the user of the apparatus to set each value when the control unit 9 performs the initialization operation.
[0084]
In the backprojection image, road shoulder portions such as guard rails, wall surfaces, hedges, sidewalk steps, and the like are expressed in a form of falling outward from the boundary position between the road and the road shoulder. In addition, since the road portion is expressed as looking down from the top, the straight road with a constant width that narrows toward the vanishing point in the image before backprojection is a road with a constant width in the backprojection image. Is expressed as
[0085]
The boundary detection unit 13 scans the back projection image RS1 to detect the boundary between the road and the road right shoulder. The boundary detection unit 13 converts the backprojection image RS1 into a horizontal region W as shown in FIG. _i2 Each area W is divided into (i = 1 to n) _i2 Scan the inside horizontally. As shown in FIG. 9B, the boundary detection unit 13 has a height W around the scanning point P2. _i2 A fixed width d set in advance at the same height as _l2 , D _r2 Region Q with _l2 , Q _r2 And set the area Q _l2 , Q _r2 Average brightness value m _Ql2 , M _Qr2 Are respectively calculated and a predetermined threshold value DD is calculated. ₁ , DD ₂ And whether the relationship of Equation 13 is satisfied is determined.
[0086]
[Formula 13]

[0087]
When the average luminance value satisfies the relationship of Equation 13, the boundary detection unit 13 sets the scanning point P2 in the backprojection image RS1 as the region W. _i2 The reliability RR represented by Equation 14 as a candidate road shoulder boundary point _P2 And is registered in a memory (not shown) built in the boundary detection unit 13.
[0088]
[Expression 14]

[0089]
The boundary detection unit 13 _i2 Every time, the reliability RR _P2 The scanning point P2 where the value of _i2 As the shoulder position on the right side of the road, the u component of the coordinate is the horizontal position of the scanning point P2, and the v component is the region W. _i2 Road shoulder representative point A as the v component of the midpoint in the height direction _i2 Are sequentially stored in a memory (not shown) built in the boundary detection unit 13. The boundary detection unit 13 stores the road shoulder position representative point A sequentially stored as shown in FIG. _i2 Is output to the control unit 9 as shoulder information on the right side of the road.
[0090]
The boundary detection unit 13 uses the relational expression (14) as the reliability for determining the shoulder position. _l2 And Q _r2 The average luminance value m of the combined area _T2 May be used as the reliability.
[0091]
[Expression 15]

[0092]
Mathematical formula 15 reliability RR _P2 When used as a reliability RR _P2 The scanning point P2 where the _i2 It can be regarded as the shoulder position.
[0093]
The backprojection unit 12 backprojects the difference image S2 obtained by the difference calculation unit 5 onto the road surface on which the vehicle is present according to the equation (12) to obtain a backprojection difference image RS2 as shown in FIG. Generate. The back projection unit 12 outputs the generated back projection difference image RS <b> 2 to the boundary detection unit 14.
[0094]
The boundary detection unit 14 scans the backprojected image RS2 to detect the boundary between the road and the road left shoulder. The boundary detection unit 14 converts the backprojection image RS2 into the horizontal region W as shown in FIG. _i1 Each area W is divided into (i = 1 to n) _i1 Scan the inside horizontally. As shown in FIG. 9A, the boundary detection unit 14 has a height W around the scanning point P1. _i1 A fixed width d set in advance at the same height as _l1 , D _r1 Region Q with _l1 , Q _r1 And set the area Q _l1 , Q _r1 Average brightness value m _Ql1 , M _Qr1 Are respectively calculated and a predetermined threshold value DD is calculated. ₁ , DD ₂ Whether or not the relationship of Equation 16 is satisfied is determined.
[0095]
[Expression 16]

[0096]
When the average luminance value satisfies the relationship of Equation 16, the boundary detection unit 14 determines the scanning point P1 in the backprojection image RS2 as the region W. _i1 And the reliability RR expressed by Equation 17 _P1 And is registered in a memory (not shown) built in the boundary detection unit 14.
[0097]
[Expression 17]

[0098]
The boundary detection unit 14 _i1 Every time, the reliability RR _P1 The scanning point P1 where the value of the _i1 As the shoulder position on the right side of the road, the u component of the coordinate is the horizontal position of the scanning point P1, and the v component is the region W. _i1 Road shoulder representative point A as the v component of the midpoint in the height direction _i1 Are sequentially stored in a memory (not shown) built in the boundary detection unit 14. The boundary detection unit 14 stores the road shoulder position representative point A sequentially stored as shown in FIG. _i1 Is output to the control unit 9 as shoulder information on the left side of the road.
[0099]
The boundary detection unit 13 uses the relational expression 17 as the reliability for determining the road shoulder position. _l1 And Q _r1 The average luminance value m of the combined area _T1 May be used as the reliability.
[0100]
[Formula 18]

[0101]
Equation 18 is used for reliability RR _P1 When used as a reliability RR _P1 The scanning point P1 where the _i1 It can be regarded as the shoulder position.
[0102]
In this embodiment as well, the configuration of the difference calculation unit and the road shoulder detection unit on the side that does not perform the detection is omitted when the road shoulder on one side only needs to be detected with respect to the traveling direction of the vehicle. It doesn't matter.
[0103]
As described above, according to the road shoulder detection device of the present embodiment, in addition to the advantages of the road shoulder detection device of the first embodiment, the scanning direction of the image at the time of road shoulder detection is the horizontal direction with respect to the screen. There is an advantage that the mounting is easy when mounting by hardware.
[0104]
(Third Embodiment) FIG. 11 is a block diagram for explaining the configuration of a vehicle control device for an automobile using the road shoulder detection device of the present invention.
[0105]
The configuration of the road shoulder detection device portion of the vehicle control device of the present embodiment is substantially the same as the road shoulder detection device described in the first and second embodiments. The difference is that the control unit 9 for performing the initialization operation and the display unit 10 for displaying the detection result are removed. As the road shoulder detection device, either the road shoulder detection device of the first embodiment or the road shoulder detection device of the second embodiment may be used.
[0106]
The output unit 15 is an interface for outputting a road shoulder detection result to the outside of the road shoulder detection unit.
[0107]
Based on the road shoulder detection result output from the output unit 15, the vehicle control unit 16 determines the position of the vehicle on which the apparatus is installed in the traveling lane, and alerts the driver when it is likely to be out of the traveling lane. It is a control part for urging and avoiding by operating the steering and brake when there is a risk of accident or collision.
[0108]
H required for shoulder detection processing at the shoulder detection device _1to2 , H _2to1 Various parameters such as the driving lane vanishing point are calculated in advance using a parameter calculation device (not shown), stored in a ROM (Read Only Memory), and incorporated into the factory before delivery. Therefore, the function for performing the initialization operation as in the first and second embodiments is unnecessary.
[0109]
As described above, the vehicle control device according to the present embodiment automatically detects a road shoulder and alerts the user when it is about to come off the road shoulder. In some cases, the avoidance operation can be automatically performed. Accidents caused by such things can be reduced.
[0110]
【The invention's effect】
According to the present invention, a region having a height different from a road surface on which a vehicle travels is extracted from a stereo image captured by two cameras, and a boundary line between the region and the road surface is obtained, thereby obtaining a stable and computationally intensive amount. It is possible to provide a few road shoulder detection apparatus and method.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a configuration of a road shoulder detection apparatus according to first and second embodiments of the present invention.
FIG. 2 is a block diagram illustrating a configuration of a road shoulder detection unit of a road shoulder detection device according to a second embodiment of the present invention.
FIG. 3 is a diagram illustrating a camera installation position in the road shoulder detection device according to the first and second embodiments of the present invention.
4A is a view for explaining an image I1 obtained from the camera 1. FIG. (B) The figure explaining the image I2 obtained from the camera 2. FIG.
FIG. 5A is a diagram illustrating a difference image S2. (B) The figure explaining difference image S1.
6A is a diagram illustrating a method for detecting a road shoulder from a difference image S2 by the road shoulder detection apparatus according to the first embodiment of the present invention. FIG. (B) The figure explaining the method of detecting a road shoulder from difference image S1 with the road shoulder detection apparatus of the 1st Embodiment of this invention.
7A is a diagram illustrating an image obtained by a camera. FIG. (B) The figure explaining a backprojection image.
FIG. 8A is a view for explaining a back-projected image RS2 of the difference image S2. (B) The figure explaining back projection image RS1 of difference image S1.
FIG. 9A is a diagram illustrating a method for detecting a road shoulder from a backprojected image RS2 by the road shoulder detection apparatus according to the second embodiment of the present invention. (B) The figure explaining the method of detecting a road shoulder from backprojection image RS1 with the road shoulder detection apparatus of the 2nd Embodiment of this invention.
10A illustrates a broken line indicating a shoulder on the camera 1 side detected by the shoulder detection device according to the second embodiment of the present invention. FIG. 10B illustrates a shoulder detection device according to the second embodiment of the present invention. The figure explaining the broken line which shows the detected shoulder of the camera 2 side.
FIG. 11 is a diagram illustrating a configuration of a vehicle control device that is a third embodiment of the present invention.
[Explanation of symbols]
1, 2 camera
3, 4 Image storage unit
5, 6 Difference calculator
7, 8 Road shoulder detector
9 Control unit
10 Display section
11, 12 Back projection unit
13, 14 Boundary detection unit
V1, V2 vanishing point
P1, P2 scanning points

Claims (8)

A method for detecting a boundary of a traveling path delimited by a three-dimensional object from a traveling vehicle,
An image acquisition step of simultaneously acquiring a first image and a second image using a first camera installed on a first side in a traveling direction of the vehicle and a second camera installed on a second side;
An image storage step of storing the first image and the second image in a memory;
Assuming that all the pixels of the second image are images of the road surface on which the vehicle is traveling, the correspondence relationship for converting the second image into a view from the first camera is used. A difference image generating step of generating a difference image by calculating an absolute value difference of luminance values between corresponding pixels of the first image and the second image;
A back projection step of back projecting the difference image onto a road surface on which the vehicle travels to generate a back projection image;
By scanning the backprojected image in the horizontal direction and obtaining a point where the change in luminance satisfies a predetermined condition, the boundary of the traveling path delimited by the three-dimensional object existing on the first side in the traveling direction is detected. A boundary detection step;
A roadside shoulder detection method. It said first side, shoulder detection process according to claim 1, characterized in that the left side in the traveling direction of the vehicle. A method for detecting a boundary of a traveling path delimited by a three-dimensional object from a traveling vehicle,
An image acquisition step of acquiring first and second images simultaneously using a second camera installed in the first camera and the right side was placed to the left side in the traveling direction of the vehicle,
An image storage step of storing the first image and the second image in a memory;
Assuming that all the pixels of the second image are images of the road surface on which the vehicle is traveling, the correspondence relationship for converting the second image into a view from the first camera is used. A road surface on which all the pixels of the first image travel the vehicle while calculating an absolute value difference between luminance values of the first image and the second image to generate a first difference image Assuming that the first image is photographed, the luminance value between the first image and the second image is calculated using a correspondence relationship for converting the first image into a view from the second camera. A difference image generation step of calculating an absolute value difference of the second difference image to generate a second difference image;
An average value of pixel values of two regions that are in contact with each other across a line segment connecting the first end point at the end of the first difference image and the vanishing point of the travel path in the difference image is determined as the first end point. Is moved along the edge of the first difference image, and a line segment connecting the first end point and the vanishing point when the average value satisfies a predetermined condition is defined as a boundary on the right side of the traveling path. and detects as the average value of each of the pixel values of the two regions in contact across a line segment and a second end point at the end of the second difference image connecting the vanishing point of the traveling path in the difference image Is obtained by moving the second end point along the end of the second difference image, and a line segment connecting the second end point and the vanishing point when the average value satisfies a predetermined condition. a boundary detection step for detecting a left boundary of the road,
A roadside shoulder detection method. A method for detecting a boundary of a traveling path delimited by a three-dimensional object from a traveling vehicle,
An image acquisition step of acquiring first and second images simultaneously using a second camera installed in the first camera and the right side was placed to the left side in the traveling direction of the vehicle,
An image storage step of storing the first image and the second image in a memory;
Assuming that all the pixels of the second image are images of the road surface on which the vehicle is traveling, the correspondence relationship for converting the second image into a view from the first camera is used. The first difference image is generated by calculating the absolute value difference of the luminance value between the first image and the second image, and all the pixels of the first image represent the road surface on which the vehicle travels. Assuming that the first image is captured, the correspondence between the first image and the second camera is converted into a view of the luminance value between the first image and the second image. A difference image generation step of calculating an absolute value difference and generating a second difference image;
A back projection step of back projecting the first difference image and the second difference image onto a road surface on which the vehicle travels to generate a first back projection image and a second back projection image;
Detecting the boundary of the road, separated by a three-dimensional object present on the left side in the traveling direction by obtaining the satisfying point changes in scanned brightness is preset to the first back projection image in the horizontal direction with the boundary of the road on which the scan to change in luminance of the second reverse projection images in the horizontal direction separated by the three-dimensional object present on the right side in the traveling direction by obtaining the satisfying point a predetermined A boundary detection step to detect;
A roadside shoulder detection method. A device that is installed in a vehicle traveling on a road and detects a boundary of a traveling path divided by a three-dimensional object,
A first camera installed on the first side of the traveling direction of the vehicle and a second camera installed on the second side;
First and second image storage units for storing first and second images taken simultaneously by the first and second cameras in units of screens;
Assuming that all the pixels of the second image are images of the road surface on which the vehicle is traveling, the correspondence relationship for converting the second image into a view from the first camera is used. A difference image generation unit that generates a difference image by calculating an absolute value difference of luminance values between corresponding pixels of the first image and the second image;
A back projection unit that back projects the difference image onto the road surface to generate a back projection image;
A boundary detection unit that detects a boundary of the travel path partitioned by the three-dimensional object on the first camera side by scanning the back-projected image in a horizontal direction to obtain a point where a change in luminance satisfies a predetermined condition. When,
A road shoulder detection device. 6. The road shoulder detection device according to claim 5, wherein the first side is a left side in a traveling direction of the vehicle. A device that is installed in a vehicle traveling on a road and detects a boundary of a traveling path divided by a three-dimensional object,
A second camera installed in the first camera and the right side was placed to the left side in the traveling direction of the vehicle,
First and second image storage units for storing first and second images taken simultaneously by the first and second cameras in units of screens;
Assuming that all the pixels of the second image are images of the road surface on which the vehicle is traveling, the correspondence relationship for converting the second image into a view from the first camera is used. , Calculating an absolute value difference of luminance values between corresponding pixels of the first image and the second image to generate a first difference image, and all the pixels of the first image include the vehicle Assuming that the road surface on which the vehicle travels is taken, the first image and the second image are converted using a correspondence relationship that converts the first image into a view from the second camera. A difference image generation unit that calculates an absolute value difference of luminance values between corresponding pixels with an image to generate a second difference image;
The average value of the pixel values of each of the two regions in contact with the line segment connecting the first end point at the end of the first difference image and the vanishing point of the travel path in the difference image is the first value. right determined while the end point is moved along the edge of the first difference image, a line segment connecting the said vanishing point and the first end point of time satisfies the said average value is determined in advance of the road along with detected as a boundary, each pixel value of said two regions in contact across a line segment and a second end point at the end of the second difference image connecting the vanishing point of the traveling path in the difference image An average value is obtained while moving the second end point along the end of the second difference image, and the second end point and the vanishing point when the average value satisfies a predetermined condition are connected. A boundary detection unit that detects a line segment as a left boundary of the road;
A road shoulder detection device. A device that is installed in a vehicle traveling on a road and detects a boundary of a traveling path divided by a three-dimensional object,
A second camera installed in the first camera and the right side was placed to the left side in the traveling direction of the vehicle,
First and second image storage units for storing first and second images taken simultaneously by the first and second cameras in units of screens;
Assuming that all the pixels of the second image are images of the road surface on which the vehicle is traveling, the correspondence relationship for converting the second image into a view from the first camera is used. , Calculating an absolute value difference of luminance values between corresponding pixels of the first image and the second image to generate a first difference image, and all the pixels of the first image include the vehicle Assuming that the road surface on which the vehicle travels is taken, the first image and the second image are converted using a correspondence relationship that converts the first image into a view from the second camera. A difference image generation unit that calculates an absolute value difference of luminance values between corresponding pixels with an image to generate a second difference image;
Back projecting the first difference image onto the road surface to generate a first back projection image, and back projecting the second difference image onto the road surface to generate a second back projection image A backprojection unit,
By obtaining a satisfying point a change has predetermined luminance by scanning the first back projection image in the horizontal direction, and detects the boundary of the road delimited by the three-dimensional object in the traveling direction of the left side , by obtaining a satisfying point a change has predetermined luminance by scanning the second backprojection image horizontally, to detect the boundaries of the road delimited by the three-dimensional object in the traveling direction of the right side A boundary detector;
A road shoulder detection device.

Images