JP2886932B2 - Facial expression recognition device - Google Patents
Facial expression recognition deviceInfo
- Publication number
- JP2886932B2 JP2886932B2 JP2050246A JP5024690A JP2886932B2 JP 2886932 B2 JP2886932 B2 JP 2886932B2 JP 2050246 A JP2050246 A JP 2050246A JP 5024690 A JP5024690 A JP 5024690A JP 2886932 B2 JP2886932 B2 JP 2886932B2
- Authority
- JP
- Japan
- Prior art keywords
- facial expression
- pattern
- expression pattern
- standard
- recognition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Landscapes
- Measurement Of The Respiration, Hearing Ability, Form, And Blood Characteristics Of Living Organisms (AREA)
- Image Processing (AREA)
Description
【発明の詳細な説明】 〔産業上の利用分野〕 本発明は，表情の時系列画像に基づき，計算機により
表情の測定を行い表情の機械認識を行う表情認識装置に
関するものである。Description: BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a facial expression recognition device that measures facial expressions by a computer based on a time-series image of facial expressions and performs machine recognition of facial expressions.
従来，例えば“工藤力訳編「表情分析入門」，発光所
誠心書房”に記されているように，心理学の分野で，
人間の視察によって人間の表情を解析する手法がいくつ
か提案されている。最近では，テレビ電話やテレビ会議
の様なシステムにおいて人間の表情や動作を送る場合
に，画像の信号の形態ではなく，「笑う」などのような
記号データとして通信するものが指向されている。昭和
63年電子入情報通信学会春季全国大会予稿Ｄ−97,内山
らの“分析合成画像符号化における顔画像の表情分析”
においては，表情の分析の１手法が提案されており，そ
れは顔の特徴点をあらかじめ指定してその変化から表情
を認識する試みである。In the field of psychology, for example, as described in “Kudo Riki Translation Edition“ Introduction to Expression Analysis ”, Hikarisho Seishin Shobo”,
Several methods for analyzing human facial expressions by human inspection have been proposed. Recently, when a human expression or motion is transmitted in a system such as a videophone or a video conference, a device that communicates as symbol data such as “laugh” instead of a signal form of an image has been used. Showa
Preliminary D-97, IEICE Spring National Convention, 1988, "Expression analysis of facial images in analysis and synthesis image coding"
Has proposed a method for analyzing facial expressions, which is an attempt to specify facial feature points in advance and recognize facial expressions from changes.
当該提案の場合には，これら特徴点の精度の高い自動
検出は困難であることと，特徴点となりうる場所が限ら
れていて必ずしも筋肉の微妙な動きまでとらえられない
という欠点があった。また，これらのデータが表情をつ
かさどる筋肉の動きとは関連性が低く，生理学的な分析
になっていないという問題点があった。In the case of this proposal, there are drawbacks that it is difficult to automatically detect these feature points with high accuracy, and that the locations that can be feature points are limited, so that it is not always possible to capture even delicate movements of muscles. In addition, there is a problem that these data have low relevance to the movements of the muscles controlling the facial expressions, and have not been analyzed physiologically.
本発明の目的は，顔面全体にわたって筋肉の微妙な動
きを計測し，その時間的変化をパターン化し，そのパタ
ーンに基づいて，表情の認識を行い精度の良い表情認識
装置を提供することにある。これにより，前記のような
画像通信のシステムにおいて表情を言葉で伝達するため
の手段を提供すると共に，コンピュータとのインタフェ
ースにおいて，人間の意志を機械に伝達する手段を提供
することが可能となる。An object of the present invention is to provide a highly accurate facial expression recognition device that measures a delicate movement of a muscle over the entire face, patterns its temporal change, and recognizes facial expressions based on the pattern. As a result, it is possible to provide a means for transmitting a facial expression in words in the above-described image communication system, and to provide a means for transmitting a human will to a machine at an interface with a computer.
本発明においては筋肉の動きを測定するモジュールを
構成部品としてそなえており，当該モジュールによっ
て，筋肉の動きの時間的変化のパターンに基づいて，表
情の認識とすることをもっとも主要な特徴とする。当該
構成部品により，従来の技術ではできなかった微妙な表
情の認識が可能となる外，しわや口の形状などにたよら
ないため，個人に依存しない表情の認識が可能である
（本モジュール「筋肉動作測定回路」の内部構成につい
ては同時に出願した「表情測定装置」で開示してい
る）。In the present invention, a module for measuring the movement of muscles is provided as a component, and the main feature of the module is to recognize facial expressions based on a temporal change pattern of muscle movement. This component makes it possible to recognize delicate facial expressions that could not be achieved by conventional technology, and also to recognize facial expressions that do not depend on individuals because they do not depend on wrinkles or the shape of the mouth. The internal configuration of the "operation measurement circuit" is disclosed in the "expression measurement device" filed at the same time.
以下，図に基づいて本発明の装置の動作を説明する。
第１図は本発明の表情認識装置全体の構成を説明する図
であって,101はテレビカメラおよびA/D変換回路等から
なる映像入力装置である。Hereinafter, the operation of the device of the present invention will be described with reference to the drawings.
FIG. 1 is a view for explaining the configuration of the entire facial expression recognition apparatus of the present invention. Reference numeral 101 denotes a video input apparatus including a television camera, an A / D conversion circuit, and the like.
表情認識装置102は，筋肉動作測定回路103,標準表情
パターン学習部105,標準表情パターン蓄積部106,表情パ
ターン計算部107,表情パターン比較部108,表情認識結果
出力部109および認識学習切り替えスイッチ104などによ
って構成される。The facial expression recognition device 102 includes a muscle movement measuring circuit 103, a standard facial expression pattern learning unit 105, a standard facial expression pattern storage unit 106, a facial expression pattern calculation unit 107, a facial expression pattern comparison unit 108, a facial expression recognition result output unit 109, and a recognition learning changeover switch 104. It is composed of
以下動作を説明する。まず，標準表情パターンの学習
モードであるとする。この場合には図中認識学習切り替
えスイッチ104をＢの側へ倒して，学習を行う。学習に
際しては，学習用の表情動作画像を映像入力装置101で
撮影して，当該得られた時系列画像を筋肉動作測定回路
103に入力して表情生成に関連する筋肉の動きを測定
し，標準表情パターン学習部105でパターンの学習を行
い，標準パターンを標準表情パターン蓄積部106に蓄積
しておく。The operation will be described below. First, it is assumed that the learning mode is a learning mode for a standard expression pattern. In this case, the learning is performed by moving the recognition learning changeover switch 104 to the side B in the figure. At the time of learning, a facial expression motion image for learning is captured by the video input device 101, and the obtained time-series image is taken by a muscle motion measuring circuit.
The standard facial expression pattern learning unit 105 learns the pattern by inputting the data to the facial expression generation unit 103 and measures the movement of the muscles related to the facial expression generation. The standard pattern is stored in the standard facial expression pattern storage unit 106.
つぎに，認識モードにおいては，認識対象の表情画像
に対して認識学習切り替えスイッチ104をＡの側に倒し
て認識を行う。認識は表情動作画像を映像入力装置101
で撮影して，当該時系列画像を筋肉動作測定回路103に
入力して表情生成に関連する筋肉の動きを測定し，これ
を表情パターン計算部107において表情パターン化し，
その表情パターンと上述のように標準表情パターン蓄積
部106に蓄積してある標準表情パターンとを表情パター
ン比較部108において比較して，類似する標準表情パタ
ーンを検索し，表情認識結果出力部109から認識対象パ
ターンが何の表情であるかの認識結果を出力する。Next, in the recognition mode, the recognition learning switch 104 is moved to the A side to perform recognition on the facial expression image to be recognized. Recognition of facial expression motion image by video input device 101
The time series image is input to the muscle movement measuring circuit 103 to measure the movement of the muscle related to the generation of the facial expression, and this is converted into a facial expression pattern by the facial expression pattern calculation unit 107.
The facial expression pattern is compared with the standard facial expression pattern stored in the standard facial expression pattern storage unit 106 as described above in the facial expression pattern comparing unit 108, and a similar standard facial expression pattern is searched. The recognition result of the expression of the recognition target pattern is output.
筋肉動作測定回路103は表情に関わる各筋肉の動きを
測定する回路で頬筋，挙筋，眼輪筋など，第２図に示す
各位置a,b,c,d,e,f,gの筋肉の動きを筋肉の方向にそっ
た成分で出力する。表情パターン計算部107は全部筋肉
の動きをベクトルデータ化する回路であって，例えば時
刻t0から時刻tnまでの頬筋の動きは（H0,H1,…,Ht,…,H
n）のようにベクトル化し，出力する。ここでHtは時刻
ｔにおける筋肉の動き量である。The muscle movement measuring circuit 103 is a circuit for measuring the movement of each muscle related to the facial expression, and measures the positions a, b, c, d, e, f, and g of each of the positions shown in FIG. Outputs the movement of the muscle as a component along the direction of the muscle. All the expression pattern calculating unit 107 is a circuit for vector data of muscle movement, for example movement of the cheek muscle from time t 0 to time t n is (H 0, H 1, ... , H t, ..., H
Vectorize as in n ) and output. Here, Ht is the amount of movement of the muscle at time t.
標準表情パターン学習部105は，学習モードの際にま
ず上記と同様のベクトルデータを生成する。笑い，怒
り，悲しみ，困惑，驚きなどのカテゴリに分けた標準表
情画像を何枚も入力して各々に上記ベクトルデータを生
成して，それぞれの表情のカテゴリごとに平均化して，
標準表情パターンとする。このほかに，学習の手法につ
いては，文字認識等で使われる様々な手法が考えられ
る。The standard expression pattern learning unit 105 first generates vector data similar to the above in the learning mode. Inputting a number of standard facial expression images divided into categories such as laughter, anger, sadness, confusion, surprise, etc., generating the above vector data for each, and averaging for each facial expression category,
Use a standard expression pattern. In addition to the above, various learning methods used in character recognition and the like can be considered.
表情パターン比較部108は入力画像に対するベクトル
パターンと標準表情パターン蓄積部106に格納してある
各表情のカテゴリに属するベクトルパターンを順番に比
較して，もっとも近いパターンのカテゴリ名とその近さ
を出力する。パターンの比較には例えばベクトル間の距
離を使う。The facial expression pattern comparing unit 108 compares the vector pattern for the input image with the vector pattern belonging to each facial expression category stored in the standard facial expression pattern storage unit 106 in order, and outputs the category name of the closest pattern and its closeness. I do. For example, the distance between vectors is used for pattern comparison.
表情認識結果は，表情パターン比較部108の結果か
ら，各カテゴリの名前，例えば「笑い」を，パターンの
近さの値から強い，弱い等の修飾後を生成し，これらを
結合して，認識した表情として出力する。As the expression recognition result, a name of each category, for example, “laughing” is generated from the result of the expression pattern comparing unit 108 after the modification of the pattern closeness, such as strong or weak, and these are combined to perform recognition. Is output as an expression.
以上説明したように，本発明によれば，顔の表情筋の
動きに基づいて，表情を非接触で認識できるから，画像
通信における表情分析モジュールとして仕えたり，心理
学や医学の分野において，微妙な筋肉の動きを非接触で
自動計測できるという利点がある。また，計算機のイン
タフェースとして人間の感情を機械に伝える機能を提供
できる。As described above, according to the present invention, the facial expression can be recognized in a non-contact manner based on the movement of the facial expression muscle, so that the present invention can serve as a facial expression analysis module in image communication, or can be subtle in the fields of psychology and medicine. There is an advantage that the movement of a muscular muscle can be automatically measured without contact. In addition, a function to transmit human emotions to the machine can be provided as a computer interface.
【図面の簡単な説明】 第１図は本発明の表情認識装置全体の構成を説明する
図，第２図は筋肉動作測定回路で計測する領域を説明す
る図である。 図中,101は映像入力装置,102は表情認識装置,103は筋肉
動作測定回路,104は認識学習切り替えスイッチ,105は標
準表情パターン学習部,106は標準表情パターン蓄積部,1
07は表情パターン計算部,108は表情パターン比較部,109
は表情認識結果出力部である。BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is a view for explaining the configuration of the entire facial expression recognition apparatus of the present invention, and FIG. 2 is a view for explaining an area measured by a muscle movement measuring circuit. In the figure, 101 is a video input device, 102 is a facial expression recognition device, 103 is a muscle movement measuring circuit, 104 is a recognition learning changeover switch, 105 is a standard facial expression pattern learning unit, 106 is a standard facial expression pattern storage unit, 1
07 is an expression pattern calculation unit, 108 is an expression pattern comparison unit, 109
Is an expression recognition result output unit.
Claims (1)
より撮影して得られた時系列画像を処理する表情認識装
置において， 当該時系列画像に対して，筋肉動作測定回路，標準表情
パターン学習部，標準表情パターン蓄積部，表情パター
ン計算部，表情パターン比較部，表情認識結果出力部を
そなえ， 前記時系列画像を前記筋肉動作測定回路に入力して表情
生成に関連する筋肉の動きを取り出し，これを前記表情
パターン計算部において表情パターン化し， 一方で，事前に標準表情の時系列画像を入力し，前記筋
肉動作測定回路で筋肉の動きを抽出した後，前記標準表
情パターン学習部でパターンの学習を行い，標準パター
ンを前記標準表情パターン蓄積部に蓄積しておき， 前記認識対象の表情パターンと前記標準表情パターン蓄
積部の標準表情パターンとを前記表情パターン比較部に
おいて比較して，類似する標準表情パターンを検索し，
前記表情認識結果出力部から認識対象パターンが何の表
情であるかの認識結果を出力する ことを特徴とする表情認識装置。A facial expression recognition device for processing a time-series image obtained by photographing a face presenting a facial expression by a video input device; It has a pattern learning section, a standard facial expression pattern storage section, a facial expression pattern calculation section, a facial expression pattern comparison section, and a facial expression recognition result output section. The facial expression pattern calculation unit converts the facial expression into a facial expression pattern. On the other hand, after inputting a time-series image of the standard facial expression in advance and extracting the muscle movement by the muscle movement measuring circuit, the standard facial expression pattern learning unit Learning the pattern in the standard facial expression pattern storage unit, and storing the standard expression pattern in the standard facial expression pattern storage unit. The quasi-facial expression pattern is compared with the quasi-facial expression pattern in the above-mentioned facial expression pattern comparing section to search for a similar standard facial expression pattern.
A facial expression recognition device, wherein the facial expression recognition result output unit outputs a recognition result indicating what facial expression the recognition target pattern is.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2050246A JP2886932B2 (en) | 1990-03-01 | 1990-03-01 | Facial expression recognition device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| JP2050246A JP2886932B2 (en) | 1990-03-01 | 1990-03-01 | Facial expression recognition device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| JPH03252775A JPH03252775A (en) | 1991-11-12 |
| JP2886932B2 true JP2886932B2 (en) | 1999-04-26 |
Family
ID=12853633
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| JP2050246A Expired - Fee Related JP2886932B2 (en) | 1990-03-01 | 1990-03-01 | Facial expression recognition device |
Country Status (1)
| Country | Link |
|---|---|
| JP (1) | JP2886932B2 (en) |
Families Citing this family (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2573126B2 (en) * | 1992-06-22 | 1997-01-22 | 正重 古川 | Expression coding and emotion discrimination device |
| US7064749B1 (en) | 1992-11-09 | 2006-06-20 | Adc Technology Inc. | Portable communicator |
| JPH06237921A (en) * | 1993-02-18 | 1994-08-30 | Matsushita Electric Ind Co Ltd | Sleeping device |
| JP2006006355A (en) | 2004-06-22 | 2006-01-12 | Sony Corp | Biological information processing apparatus and video / audio reproduction apparatus |
| JP5574407B2 (en) * | 2010-01-14 | 2014-08-20 | 国立大学法人 筑波大学 | Facial motion estimation apparatus and facial motion estimation method |
| JP6860720B2 (en) * | 2020-02-28 | 2021-04-21 | 株式会社トプコン | Ophthalmic examination equipment |
-
1990
- 1990-03-01 JP JP2050246A patent/JP2886932B2/en not_active Expired - Fee Related
Also Published As
| Publication number | Publication date |
|---|---|
| JPH03252775A (en) | 1991-11-12 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20250363350A1 (en) | Method and system for activity classification | |
| Owayjan et al. | The design and development of a lie detection system using facial micro-expressions | |
| KR102195401B1 (en) | hand language recognition method for hearing impairment people | |
| US11475711B2 (en) | Judgement method, judgement apparatus, and recording medium | |
| KR102925248B1 (en) | Apparatus and method for analyzing motion similarity | |
| KR20210018028A (en) | Handwriting and arm movement learning-based sign language translation system and method | |
| JP2886932B2 (en) | Facial expression recognition device | |
| JP5846552B2 (en) | Sign language recognition method and apparatus | |
| Cao et al. | Human posture recognition using skeleton and depth information | |
| Büchner et al. | Let's Get the FACS Straight--Reconstructing Obstructed Facial Features | |
| CN113033501A (en) | Human body classification method and device based on joint quaternion | |
| Anwar et al. | Real time intention recognition | |
| Bin et al. | High performance face recognition system by creating virtual sample | |
| JP3098276B2 (en) | Facial expression recognition device | |
| Li et al. | CAS-AIR-3D face: A low-quality, multi-modal and multi-pose 3D face database | |
| Murase et al. | Parametric eigenspace representation for visual learning and recognition | |
| CN119445682A (en) | Emotion recognition method, device, electronic device and storage medium | |
| Ghosh et al. | R-SLR: Real-Time Sign Language Recognition System | |
| Imoize et al. | Development of an infrared-based sensor for finger movement detection | |
| Tian et al. | Automatic detecting neutral face for face authentication and facial expression analysis | |
| Ayyalasomayajula et al. | A multimodal deep learning approach to emotion detection and identification | |
| Hagimoto et al. | A Dynamic Finger Character Recognition Method Using Landmark Behavior Rule Base | |
| JP2886933B2 (en) | Facial expression measuring device | |
| Hienz et al. | Automatic estimation of body regions from video images | |
| Bekhouch et al. | Few Skeleton Features For Gate-Based Gender Recognition From Landmark Frames |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| LAPS | Cancellation because of no payment of annual fees |