JP3715435B2 - Information processing method, information processing apparatus, and recording medium - Google Patents

Description

と記述する。ラベル「Ｐｒｏｔｅｃｔ」を仮想物体の記述データの所定の位置に挿入することにより、この仮想物体の記述データを処理する情報加工装置のアニメーション制御部４は、加工した結果を形状データ記憶部８に保存する際、まず、バックグラウンドで再生し、それを単純な動画データ（たとえば、ＡＶＩファイルなど）として保存する。このようにすれば、ユーザは仮想物体の加工を楽しめ、かつ仮想物体の記述データに対するその提供者の権利（例えば著作権）を保護することができる。
【００５０】
以上説明したように、上記第１の実施形態によれば、ユーザは選択したチャネルの音を聞きながら、３次元ＣＧモデルなどのキャラクタへの振り付けを行うことができるので、誰もがわかりやすく簡単に、ＣＧモデルに対する知識がなくても、アニメーションの作成を行うことができる。
（第２の実施形態）
図７は、本発明の第２の実施形態に係る情報加工装置の構成を概略的に示したものである。なお、図１と同一部分には同一符号を付し、異なる部分について説明する。すなわち、図７の情報加工装置には、図１に示した構成部に、さらに、動き入力部１０が追加されている。
【００５１】
動き入力部１０は、主に、画像取得部１０ａと画像処理部１０ｂとから構成され、画像処理部１０ｂは、画像取得部１０ａで取得された画像から撮像物の動きを抽出するものである。
【００５２】
動き入力部１０で抽出された撮像物の動き情報は、例えば、アニメーション制御部４に送られて、ここで、当該動き情報に基づき仮想物体を動かす。また、当該動き情報に基づき動き要素編集部５で動き要素を作成して、それを動き要素記憶部６に記憶する。
【００５３】
動き入力部１０で、撮像対象の動き（たとえば人間手の動きや体の動き）を取得するには、たとえば、画像取得部１０ａとしてＣＣＤカメラを用いて、通常のビデオ画像から画像処理により、手あるいは体を背景より抜き出し、さらに抜き出した画像から動きを抽出するようにしてもよい。
【００５４】
画像取得部１０ａは、反射光を受光し、距離画像を取得する、例えば特願平９−２９９６４８号に記載されているような距離画像取得装置であってもよい。以下、画像取得部１０ａが距離画像取得装置である場合を例にとり説明する。
【００５５】
ここで、距離画像を取得する画像取得部１０ａ（以下、距離画像取得部１０ａと呼ぶ）および距離画像について簡単に説明する。
距離画像取得部１０ａの外観を図８に示す。中央部には円形レンズとその後部にあるエリアセンサ（図示せず）から構成される受光部１０３が配置され、円形レンズの周囲にはその輪郭に沿って、赤外線などの光を照射するＬＥＤから構成される発光部１０１が複数個（例えば８個）等間隔に配置されている。
【００５６】
発光部１０１から照射された光が物体に反射され、受光部１０３のレンズにより集光され、レンズの後部にあるエリアセンサで受光される。エリアセンサは、例えば２５６×２５６のマトリックス状に配列されたセンサで、マトリックス中の各センサにて受光された反射光の強度がそれぞれ画素値となる。このようにして取得された画像が、図１０に示すような反射光の強度分布としての距離画像である。
【００５７】
図９は、距離画像取得部１０ａの構成例を示したもので、主に、発光部１０２、受光部１０３、反射光抽出部１０２、タイミング信号生成部１０４から構成される。
【００５８】
発光部１０１は、タイミング信号生成部１０４にて生成されたタイミング信号に従って時間的に強度変動する光を発光する。この光は発光部前方にある対象物体に照射される。
【００５９】
受光部１０３は、発光部１０１が発した光の対象物体による反射光の量を検出する。
反射光抽出部１０２は、受光部１０３にて受光された反射光の空間的な強度分布を抽出する。この反射光の空間的な強度分布は画像として捉えることができるので、以下、これを距離画像と呼ぶ。
【００６０】
受光部１０３は一般的に発光部１０１から発せられる光の対象物による反射光だけでなく、照明光や太陽光などの外光も同時に受光する。そこで、反射光抽出部１０２は発光部１０１が発光しているときに受光した光の量と、発光部１０１が発光していないときに受光した光の量の差をとることによって、発光部１０１からの光の対象物体による反射光成分だけを取り出す。
【００６１】
反射光抽出部１０２では、受光部１０３にて受光された反射光から、その強度分布、すなわち、図１０に示すような距離画像を抽出する。
図１０では、簡単のため、２５６×２５６画素の距離画像の一部である８×８画素の距離画像の場合について示している。
【００６２】
物体からの反射光は、物体の距離が大きくなるにつれ大幅に減少する。物体の表面が一様に光を散乱する場合、距離画像１画素あたりの受光量は物体までの距離の２乗に反比例して小さくなる。
【００６３】
図１０において、行列中のセルの値（画素値）は、取得した反射光の強さを２５６階調（８ビット）で示したものである。例えば、「２５５」の値があるセルは、距離画像取得部１０ａに最も接近した状態、「０」の値があるセルは、距離画像取得部１０ａから遠くにあり、反射光が距離画像取得部１０ａにまで到達しないことを示している。
【００６４】
距離画像の各画素値は、その画素に対応する単位受光部で受光した反射光の量を表す。反射光は、物体の性質（光を鏡面反射する、散乱する、吸収する、など）、物体の向き、物体の距離などに影響されるが、物体全体が一様に光を散乱する物体である場合、その反射光量は物体までの距離と密接な関係を持つ。手などは、このような性質をもつため、距離画像取得部１の前方に手を差し出した場合の距離画像は、手までの距離、手の傾き（部分的に距離が異なる）などを反映する図１１に示したような３次元的なイメージを得ることができる。
【００６５】
動き入力部１０の画像処理部１０ｂは、図１０に示したような形式ので距離画データに対し、エッジ切り出し（撮像物体の輪郭抽出）、重心抽出、面積算出、撮像物体までの距離の算出、動きベクトルの算出など、撮像体の形状、動き、位置等の情報を抽出するための種々の画像処理を行う。
【００６６】
画像処理部１０ｂでは、まず、図１０に示したような距離画像データに対し、画素値が予め定められた所定値以下のセルを除き、例えば、図１２に示すように当該距離画像内にある撮像された物体の輪郭情報を抽出する。
【００６７】
図１２のような輪郭情報を抽出するには、隣り合う画素の画素値を比較し、画素値が一定値α以上のところだけに定数値を入れて、同じ定数値が割り振られた連続した画像領域の画素を抽出すればよい。
【００６８】
すなわち、マトリックス上の座標位置（ｉ、ｊ）にある画素値をＰ（ｉ、ｊ）とし、輪郭情報の画素値をＲ（ｉ、ｊ）とすると、

とすることにより、図１２のような物体の輪郭情報を得ることができる。
【００６９】
各距離画像から抽出された図１２に示したような輪郭情報を基に、当該物体の重心を計算する。
この重心位置と直前の距離画像から抽出された重心位置とを比較して物体が動いているかどうかが判断できる。動いていると判断したときは、重心の変化量と変化方向とを表す動きベクトルを算出する。また、重心位置のみに限らず、連続する距離画像のそれぞれから抽出された輪郭情報を互いに比較しあって、輪郭上の任意の点の動きベクトルを求めてもよい。
【００７０】
輪郭の抽出された物体までの距離ｄの算定には、まず当該物体の重心位置近傍の画像の代表画素値を求める。代表画素値としては、平均値、最近傍値などいくつかあるが、ここでは、最近傍値を使うとする。当該物体からの反射光の強さは物体までの距離の２乗に反比例して小さくなる。すなわち、当該物体の画像の代表画素値をＱ（ｉ、ｊ）とすると、
Ｑ（ｉ、ｊ）＝Ｋ／ｄ² …（１）
と表すことができる。
【００７１】
ここで、Ｋは、例えば、ｄ＝０．５ｍのときに、画素値Ｒ（ｉ、ｊ）の値が「２５５」になるように調整された係数である。式（１）をｄについて解くことで、距離値ｄを求めることができる。
【００７２】
このようにして、距離画像のそれぞれから目的とする対象物の形状、動き、位置等の情報を抽出することができる。
図１３は、図７の情報加工装置の処理動作を説明するためのフローチャートである。なお、図２と同一部分には同一符号を付し、異なる部分について説明する。すなわち、図２のステップＳ６〜ステップＳ７の仮想物体に動きを対応付ける処理が、図１３ではステップＳ２１〜ステップＳ２３の処理に置き換わっている。
【００７３】
仮想物体が選択された時点で、呈示部７は、例えば、図１４に示すような、動きの編集ウインドウを表示する。
図１４に示す動き編集ウインドウには、図３と同様に、当該仮想物体の形状を表示する領域ｗ２、部品選択メニューバーｗ５が表示されている。例えば、腰を選択したとすると、動き編集ウインドウには、ユーザの動き（例えば、手の動き）を入力するためのインタフェースとして、動き入力ウインドウｗ１４および距離画像取得部１０ａによる距離画像の開始を指示するための「動き入力スタート」ボタンＢ１１および距離画像取得部１０ａによる距離画像の終了を指示するための「動き入力ストップ」ボタンＢ１２と、先に選択されたＭＩＤＩファイルから抽出される音楽パート（チャネル）の選択領域ｗ３とを表示する。
【００７４】
ステップＳ３の待ち状態から、ユーザが「動き入力スタート」ボタンＢ１１を選択すると（ステップＳ２１）、距離画像取得部１０ａが撮像対象のユーザの手の距離画像の取得を開始する（ステップＳ２２）。ユーザが図９に示したような距離画像取得部１０ａの所定の撮像範囲内で、たとえば手を動かすと、その動きを距離画像取得部１０ａが例えば毎秒３０枚程度のレートで撮像し、撮像した距離画像を動き入力ウインドウｗ１４に表示する。それと同時に画像処理部１０ｂは、前述したように、取得した距離画像に対し、エッジ切り出し（撮像物体の輪郭抽出）、重心抽出、面積算出、撮像物体までの距離の算出、動きベクトルの算出などの画像処理を行って、撮像体（ユーザの手）の形状、動き、位置等の情報を抽出する。この抽出された動き情報に基づきアニメーション制御部４は、先に選択されたＭＩＤＩファイルのチャネル「１」の音の高低にあわせて、選択されている「腰」の部品の動きの描画を呈示部７に指示する（ステップＳ２３）。
【００７５】
前述の第１の実施形態では、仮想物体に動きを付けるときには、動き編集ウインドウの動き選択メニューｗ４に表示された詳細項目の組み合わせによっていた。すなわち、移動は「左右に移動」「上下に移動」「前後に移動」の中のどれか、回転は「縦に回転」「横に回転」「手前に回転」のどれか、拡大縮小は、「横に拡大縮小」「縦に拡大縮小」「全体に拡大縮小」のいずれかというように、動きを１つずつ選択して組み合わせて行く必要があった。が、第２の実施形態では、ユーザの手や体の動きによって直接入力できる。たとえば、手を縦に回転させながら、左右に移動すれば、一度に回転と移動を入力できるのである。
【００７６】
画像処理部１０ｂは、距離画像から抽出される撮像対象（例えば手）の輪郭の重心位置の移動ベクトルと動きの詳細項目（例えば、移動であれば「左右に移動」「上下に移動」「前後に移動」、回転であれば「縦に回転」「横に回転」「手前に回転」のどれか、拡大縮小であれば「横に拡大縮小」「縦に拡大縮小」「全体に拡大縮小」）との対応関係を予め記憶しているものとする。そして、実際に取得した距離画像から抽出された撮像対象の輪郭の重心位置の移動ベクトルに該対応関係を照らし合わせてみて、該撮像対象の動きを検知するとしてもよい。すなわち、距離画像から抽出された移動ベクトルが、どのような動きに対応する移動ベクトルの合成であるか等に基づき、当該撮像対象の動きを判断してもよい。
【００７７】
また、距離画像から抽出された撮像対象の動きをより詳細に解析すれば（例えば、移動ベクトルを撮像対象の輪郭上の任意の点から求める）、距離画像から抽出された撮像対象（例えば手）の動きに対応した細やかな動きを仮想物体に対応付けることができるであろう。
【００７８】
動きの入力は、一定時間がすぎると、終了し、再び、入力状態になる。その時は、画面には、自分の手や体の画像が表示されていないので、動きの入力はできない。
【００７９】
以上説明したように、上記第２の実施形態によれば、ユーザは自分の手や体を使って、自然に動きを入力できるので、直感的にアニメーションを作成できる。特に、幼児向けの体を使った音楽教育にも適用できる。
（第３の実施形態）
図１５は、本発明の第３の実施形態に係る情報加工装置の構成を概略的に示したものである。なお、図１と同一部分には同一符号を付し、異なる部分について説明する。すなわち、図１５の情報加工装置には、図１に示した構成部に、さらに、ユーザにより作成されたアニメーションの動きを評価する評価部１１が追加されている。
【００８０】
評価部１１は、動きを評価してその結果を、たとえば点数などで算出するものである。例えば、演歌、歌謡曲、単語、ワルツといった音楽の種別に応じて予め各動き（移動、回転、拡大縮小）毎にそれぞれ重み値をｗｍ、ｗｒ、ｗｃと定めておき、仮想物体の各部品（ｉ）毎に、ユーザにより対応付けられた動きの量、すなわち、移動量ｍｉ、回転量ｒｉ、変形量ｃｉを算出して、次式から評価値Ｅを求める。
【００８１】
【数１】

【００８２】
評価部１１で算出された評価値は、呈示部７に呈示されたり、あるいは、評価値Ｅが、例えば、ある一定値を超えていれば、「Ｉｔ’ｓ ｃｏｏｌ」などのほめるメッセージを呈示部７に呈示してもよい。さらに、本発明の情報加工装置をアーケードゲームなどのゲーム機に適用して、評価値Ｅが一定値を超えると景品等を出すようにしてもよい。
【００８３】
単に評価値をユーザに呈示するのみに限らず、評価値により、複数のユーザ間で勝敗を競うようにすることも可能である。また評価の形式も上記の方法に必ずしも限定するものではない。
【００８４】
【発明の効果】
以上説明したように、本発明によれば、３次元あるいは２次元仮想空間内の仮想物体の記述に音楽や動きを付けてアニメーションの作成等の情報の加工および利用が容易に行える。
【図面の簡単な説明】
【図１】本発明の第１の実施形態に係る情報加工装置の構成を概略的に示した図。
【図２】図１の情報加工装置の処理動作を説明するためのフローチャート。
【図３】呈示部に呈示される動きの編集ウインドウの一例を示した図。
【図４】ラベル記述の挿入されたＶＲＭＬ文書の一例を示した図。
【図５】ラベル記述の挿入されたＶＲＭＬ文書の一例を示した図。
【図６】ラベル記述の挿入されたＶＲＭＬ文書の一例を示した図。
【図７】本発明の第２の実施形態に係る情報加工装置の構成を概略的に示した図。
【図８】距離画像取得部の外観の一例を示した図。
【図９】距離画像取得部の構成例を示した図。
【図１０】反射光の強度を画素値とする距離画像の一例を示した図。
【図１１】図１０に示した様なマトリックス形式の距離画像を３次元的な表した図。
【図１２】距離画像から抽出された物体の輪郭画像の一例を示した図。
【図１３】図７の情報加工装置の処理動作を説明するためのフローチャート。
【図１４】呈示部に呈示される動きの編集ウインドウの他の例を示した図。
【図１５】本発明の第３の実施形態に係る情報加工装置の構成を概略的に示した図。
【符号の説明】
１…音楽情報入力部
２…音楽情報抽出部
３…音楽情報再生部
４…アニメーション制御部
５…動き要素編集部
６…動き要素記憶部
７…呈示部
８…形状データ記憶部
９…入力部
１０…動き入力部
１０ａ…（距離）画像取得部
１０ｂ…画像処理部
１１…評価部[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an information processing apparatus that can interactively process and use information such as creation of animation by adding music and movement to three-dimensional CG (computer graphics) data interactively.
[0002]
[Prior art]
With the spread of personal computers and improved performance, it has become easier for individuals to enjoy and operate computer music and computer graphics. However, in practice, since a computer graphics creation tool is difficult to operate, not everyone can easily create moving computer graphics (animation).
[0003]
As an apparatus for creating computer art based on sound and video, there is an apparatus that determines graphics information from performance information of a concert (Japanese Patent Laid-Open No. 6-110452). The present invention creates computer art by automatically determining the position, color, shape, size, orientation and speed of change of figures from performance information of a digital sound source such as MIDI (Music Instrument Digital Interface). It is.
[0004]
However, the relationship between music information, graphics, and motion information is fixed by a calculation formula or a correspondence table prepared in advance, and one piece of music is not changed unless the performance method is changed. Only video could be generated.
[0005]
Also, if the relationship is changed or added / deleted according to the intention of the creator, editing cannot be performed. Also, the relationship edited by the creator could not be recorded.
[0006]
[Problems to be solved by the invention]
As described above, conventionally, there has been a problem that it is difficult to process and use information such as creation of animation by adding music or movement to virtual object data in a three-dimensional or two-dimensional virtual space.
[0007]
Accordingly, an object of the present invention is to provide an information processing method that can easily process and use information such as creation of animation by adding music and movement to virtual object data in a three-dimensional or two-dimensional virtual space. .
[0008]
It is another object of the present invention to provide an information processing apparatus that can easily process and use information such as creation of animation by adding music and movement to virtual object data in a three-dimensional or two-dimensional virtual space. .
[0009]
[Means for Solving the Problems]
(1) The information processing method of the present invention extracts sound components (for example, in the case of MIDI, instrument part (channel), pitch of each channel, etc.) from input sound data, and is stored in advance. In the description of the virtual object selected from the description of the virtual object in the virtual space (for example, VRML document), the description of the action selected from the action descriptions of the virtual object stored in advance and the extracted sound The component of the selected sound is associated with the selected sound component (for example, a VRML label description is inserted), and the description of the selected virtual object is processed to correspond to the selected virtual object. By presenting attached motion and sound components, it is possible to process and use information such as creating animation by adding music and motion to the description of a virtual object in a 3D or 2D virtual space. It can be performed to. In addition, when selecting a sound component and a motion to be associated with a virtual object, the selected motion and sound component are presented at the same time, so that it is easy to check the information processing result.
[0010]
(2) The information processing method of the present invention extracts a sound component from input sound data, acquires a captured image of a recognition target, recognizes the movement of the recognition target from the image, The description of the virtual object selected from the description of the virtual object in the stored virtual space is selected from the description of the action corresponding to the recognized movement of the recognition target and the extracted sound component. 3D by processing the description of the selected virtual object by associating the selected sound object with the selected sound object, and presenting the movement and sound component associated with the selected virtual object. Alternatively, it is possible to easily process and use information such as creation of animation by adding music and movement to the description of the virtual object in the two-dimensional virtual space. In addition, when selecting a sound component and a motion to be associated with a virtual object, the selected motion and sound component are presented at the same time, so that it is easy to check the information processing result.
[0011]
(3) The information processing apparatus of the present invention includes a first storage unit that stores a description of a virtual object in a virtual space,
Second storage means for storing a description of the motion of the virtual object;
Extraction means for extracting sound components from the input sound data;
The description of the selected virtual object among the virtual objects stored in the first storage means is extracted from the description of the action stored in the second storage means and extracted by the extraction means. Processing means for processing the description of the selected virtual object by associating the selected sound component among the selected sound components;
Presenting means for presenting movement and sound components associated with the selected virtual object;
With this, it is possible to easily process and use information such as creation of animation by adding music and movement to the description of the virtual object in the three-dimensional or two-dimensional virtual space. In addition, when selecting a sound component and a motion to be associated with a virtual object, the selected motion and sound component are presented at the same time, so that it is easy to check the information processing result.
[0012]
(4) An information processing apparatus according to the present invention includes a storage unit that stores a description of a virtual object in a virtual space;
Motion recognition means for acquiring a captured image of a recognition target and recognizing the movement of the recognition target from the image;
Extraction means for extracting sound components from the input sound data;
The description of the virtual object selected from the virtual objects stored in the storage means, the description of the action corresponding to the motion of the recognition target recognized by the motion recognition means, and the components of the sound extracted by the extraction means Processing means for processing the description of the selected virtual object by associating the selected sound component with
Presenting means for presenting movement and sound components associated with the selected virtual object;
With this, it is possible to easily process and use information such as creation of animation by adding music and movement to the description of the virtual object in the three-dimensional or two-dimensional virtual space. In addition, when selecting a sound component and a motion to be associated with a virtual object, the selected motion and sound component are presented at the same time, so that it is easy to check the information processing result.
[0013]
DETAILED DESCRIPTION OF THE INVENTION
Embodiments of the present invention will be described below with reference to the drawings.
(First embodiment)
FIG. 1 schematically shows the configuration of an information processing apparatus according to the first embodiment of the present invention. A music information input unit 1, a music information extraction unit 2, a music information playback unit 3, and an animation control unit 4. , A movement element editing unit 5, a movement element storage unit 6, a presentation unit 7, a shape data storage unit 8, and an input unit 9.
[0014]
The music information input unit 1 stores music information such as MIDI (Musical Instrument Digital Interface) (hereinafter, described as an example of a MIDI file) on a recording medium such as a CD-ROM or the Internet. input.
[0015]
The music information extraction unit 2 extracts the musical instrument part (for example, a channel in MIDI) and the sound parameter of each musical instrument part (in the case of MIDI, the level of the sound in each channel) from the music information input from the music information input unit 1. Etc.).
[0016]
The music information reproduction unit 3 reproduces all the musical instrument parts extracted by the music information extraction unit 2 or only designated parts.
The shape data storage unit 8 stores, for example, shape description data of a three-dimensional CG model (virtual object) described in a format such as VRML (Virtual Reality Modeling Language) (hereinafter, virtual object shape description data described in VRML). Are sometimes referred to as VRML documents).
[0017]
The movement element storage unit 6 includes movement elements (for example, periodic movements such as parallel movement, rotation movement, and elastic movement in the three-dimensional virtual space) that are added to the description data of the virtual object stored in the shape data storage section 8. A file of description data of actions such as simple exercises).
[0018]
The motion element editing unit 5 extracts the motion element stored in the motion element storage unit 6 and the music information extraction unit 2 into the description data of the selected virtual object among the virtual objects stored in the shape data storage unit 8. Combined and matched with the instrument part. That is, in the motion element editing unit 5, the VRML document as the virtual object description data stored in the shape data storage unit 8 is not the file describing the motion selected by the user or the music data file itself, A label to be represented (for example, a conventional VRML label description) is inserted (see FIGS. 4 to 6). This label description part is skipped when a VRML document is interpreted by a normal VRML browser, and can be interpreted only by the information processing apparatus of the present invention. When the VRML document with the label read from the shape data storage unit 8 is actually reproduced, the inserted label is replaced with the description content of the corresponding file.
[0019]
The animation control unit 4 actually moves the virtual object on the display screen of the presentation unit 7 based on the motion element associated (selected) with the description data of the virtual object by the motion element editing unit 5. Is drawn as
[0020]
The presenting unit 7 displays a predetermined screen on the display device.
The input unit 9 inputs instructions from the user from various input devices such as a mouse and a keyboard.
[0021]
From the information processing apparatus having such a configuration, the user moves to the desired virtual object among the virtual objects stored in the shape data storage unit 8 according to the desired instrument part extracted by the music information extraction unit 2. By associating the movement elements stored in the element storage unit 6, for example, the virtual object can be choreographed for dance according to music.
[0022]
FIG. 2 is a flowchart for explaining the processing operation of the information processing apparatus of FIG. In practice, the input is interpreted, and the process branches according to the result, which is more complicated than the process flow shown in FIG. However, here, in order to facilitate the explanation, parts other than those related to the gist of the present embodiment are omitted. The MIDI file to be choreographed is, for example, recorded on a recording medium, and a MIDI file having a title selected in advance by the user is read from the recording medium and input to the music information input unit 1.
[0023]
First, a virtual object to be moved is selected via a virtual object (three-dimensional data) selection window displayed on the display screen of the presenting unit 7 (step S1).
For example, in the virtual object selection window, icons displaying the outline of the virtual object stored in the shape data storage unit 8 are listed, and by clicking one of them with the mouse, a desired object is displayed. A virtual object is selected.
[0024]
When the virtual object is selected, the presentation unit 7 displays a motion editing window as shown in FIG. 3, for example.
The motion editing window shown in FIG. 3 has a region w2 for displaying the shape of the virtual object, and an overall image of the previously selected virtual object is displayed here. Here, the selected virtual object is a virtual object in the shape of a robot, and can move the entire virtual object and parts such as the waist, chest, head, right hand, left hand, right foot, and left foot. To do. For example, the user selects a part to be choreographed by clicking the desired part of the robot displayed in the display area w2 with the mouse or by clicking the desired part from the part selection menu bar w5 with the mouse. (Step S2). For example, if waist is selected, the motion editing window is extracted from the motion selection menu w4 displayed in a list so that the motions that can be associated with the waist can be selected, and the previously selected MIDI file. A music part (channel) selection area w3 is displayed.
[0025]
In the movement selection menu w4, detailed items of movement are displayed for each conceptual large item such as “movement”, “rotation”, and “enlargement / reduction”, and a radio button is provided on the left side of each detailed item. Each large item is not selected or one detailed item can be selected. For example, the large item “move” includes detailed items “move left and right”, “move up and down”, and “move back and forth”, and one of these detailed items can be selected. The large item “Rotate” includes detailed items “Rotate vertically”, “Rotate horizontally”, and “Rotate forward”, and one of these detailed items can be selected. The large item “enlargement / reduction” includes detailed items “horizontal enlargement / reduction”, “vertical enlargement / reduction”, and “enlargement / reduction”, and one of these detailed items can be selected. The motion elements (motion description data) corresponding to the detailed items are stored in the motion element storage unit 6.
[0026]
In FIG. 3, the part of the selected virtual object, that is, “waist” is specially displayed (for example, highlighted display, reverse display, blinking display) on the display area w2 of the virtual object. Further, the motion editing window is provided with a “no motion” button B1, an “OK” button B2, a “cancel” button B3, and an “update” B4 which are selected when no movement is applied to the selected virtual object part. Yes.
[0027]
In a state where the motion editing window as shown in FIG. 3 is displayed, an input waiting state is entered (step S3).
In the channel selection area w3, “0” is displayed before the channel is selected. This means that no channel has been selected. In this state, since the channel is not selected, the MIDI file of the title selected previously is reproduced by the music information reproducing unit 3.
[0028]
When the user selects a desired channel by clicking the button indicated by the left-pointing and right-pointing arrows in the channel selection area w3 from the waiting state in step S3 (step S4), the selected channel, For example, only the channel “1” is reproduced by the music information reproducing unit 3 (step S5). Thereafter, the process waits again (step S3).
[0029]
From the waiting state in step S3, the user selects a desired detailed item from the detailed items displayed in the movement selection window w4 (step S6). In FIG. 3, “Rotate horizontally” is selected. Then, the animation control unit 4 reads out a description file for “rotating horizontally” the virtual object from the motion element storage unit 6, and adjusts it to the level of the sound of the channel “1” of the previously selected MIDI file. The presentation unit 7 is instructed to draw a motion of rotating the selected “waist” part horizontally. That is, the waist of the virtual object displayed in the display area w2 starts to move in accordance with the currently reproduced channel “3” (step S7).
[0030]
Note that “moving”, “rotating”, and “enlarging / reducing” can be simultaneously added to the same component. Any one detail item in "Move", any one detail item in "Rotate", and any one detail item in "Enlarge / Reduce" can be selected for one part at the same time it can.
[0031]
If one detailed item is selected, the process returns to step S3 and waits for input. If a detailed item is selected subsequently, the process proceeds to step S6, and the movement can be choreographed in the same manner. Alternatively, the user can change his mind and proceed to step S4 to select a channel other than the already selected channel, or proceed to step S8 to select a different part and perform choreography in the same manner as described above. .
[0032]
In the motion editing window shown in FIG. 3, the music information and the motion element are selected for each part of the virtual object. However, the music information and the motion element can be selected for the entire virtual object. For example, by selecting “whole” from the component selection menu bar w5 in the motion editing window of FIG. 3, a motion selection menu displaying a list of motions that can be associated with the entire virtual object and a channel selection region w3 are displayed. Then, the desired item of the desired motion is selected from the motion selection menu, and the desired channel may be selected in the channel selection area w3.
[0033]
As described above, after selecting a desired instrument part and movement for each component of the virtual object selected by the user and performing choreography, the “OK” button on the movement editing window in FIG. B2, when the “update” button B4 is selected (step S9), the animation control unit 4 creates a VRML document in which the label description as shown in FIGS. 4 to 6 created by the motion element editing unit 5 is inserted. Is stored in the shape data storage unit 8 (step S10).
[0034]
When the “cancel” button B3 is selected from the waiting state in step S3 (step S11), the label description in the VRML document originally stored in the shape data storage unit 8 is not changed and is newly selected. The performed movement is invalidated (default state) (step S12).
[0035]
When the “no motion” button B1 is selected from the waiting state in step S3 (step S13), the motion label description in the VRML document originally stored in the shape data storage unit 8 is deleted (step S14). ).
[0036]
When the box displaying “X” in the upper right of the motion editing screen shown in FIG. 3 is clicked with the mouse, it becomes “editing end instruction”, the motion editing window is closed, and the processing is ended (step S14). ).
[0037]
When the animation created as described above is reproduced, the animation control unit 4 reads out the VRML document in which the label description is inserted from the shape data storage unit 8 and draws about 30 frames per second by the presentation unit 7. This makes it appear to the human eye that the virtual object is moving.
[0038]
Next, the VRML document in which the label description created by the motion editing unit 5 is inserted will be described. As described above, the VRML document in which the label description is inserted is stored in the shape data storage unit 8 in accordance with an instruction from the user upon completion of motion editing.
[0039]
The shape data storage unit 8 stores, for example, VRML documents into which motion elements and music information label descriptions as shown in FIGS. 4 to 6 are inserted. Here, for the sake of simplicity, an example is shown in which a virtual object composed of five parts (a truncated cone (1), a sphere (2), and a cylinder (2)) is shown in the description of VRML.
[0040]
The description “Label {}” indicates that there is a structure indicating a label in {}. The first label description “label“ MIDI ”” indicates that a MIDI file is used. The following label description “label” sakura2. “mid” ”indicates a MIDI file name to be used. In this case, it is indicated that a MIDI file saved with the name “sakura2.mid” is used. “Mid” at the end of the file name indicates that the file type is a MIDI file.
[0041]
The label description part is simply a name in a normal browser (such as WebSpace) that interprets VRML, and is skipped without being interpreted. Therefore, even after being processed by the information processing apparatus of the present embodiment, a portion handled by a normal browser can be displayed without change. Conversely, whether or not the label description part is created by the information processing apparatus of this embodiment by checking whether there is a description of a specific label used in the information processing apparatus of this embodiment, for example, MIDI. Can be determined.
[0042]
The description “Separator {}” indicates the presence of an object part (part) in the three-dimensional space in {}. In the example of FIG. 4, the first “Separator” is a part indicating how to correspond to the MIDI file. Since the contents of this part are all described using labels, other browsers simply skip them. The first label “transX” indicates how to move in response to a MIDI sound (pitch, rhythm, etc., for example, a pitch here). For example, there are the following types of labels representing the movement of the sound, which correspond to the detailed items in the movement selection window w4 displayed on the movement editing screen of FIG. 3 and are selected by the user. In accordance with the detailed items, the motion element editing unit 5 selects from the following labels and inserts them into the VRML document.
[0043]
transX: Translation in the X-axis direction
transY: translation in the Y-axis direction
transZ: Translation in the Z-axis direction
rotX: rotational movement around the X axis
rotY: rotational movement around the Y axis
rotZ: rotational movement around the Z axis
scaleX: Scale change in the X-axis direction
scaleY: Changing the scale in the Y-axis direction
scaleZ: Scale change in the Z-axis direction
The next label “Channel 2” indicates which MIDI channel to be played corresponds to the movement described by the label above it. Here, “Channel 2” is designated. According to the channel selected by the user in the music part (channel) selection area w3 of the motion editing screen of FIG. 3, the motion element editing unit 5 selects a label and inserts it into the VRML document.
[0044]
The next label “elastic” indicates whether or not to add a motion of stretching or contracting as an elastic body. When expanding and contracting as an elastic body, it is possible to expand and contract in accordance with sound. However, if the change of the pitch is large, it will be excessively expanded and contracted. That is, there are the following two types of labels.
[0045]
elastic: stretches as an elastic body
noneDE: Does not stretch
The motion selection window w4 displayed on the motion editing screen of FIG. 3 does not include a detailed item for selecting a motion as an elastic body, but if such a detailed item can be selected, it is selected by the user. In accordance with the detailed item, the motion element editing unit 5 selects one of the two labels and inserts it into the VRML document.
[0046]
The next two labels “damy1” and “damy2” are reserved for expansion.
Since the correspondence between the motion and the channel described here and the presence / absence of expansion / contraction are first, it is effective for the entire object composed of the parts described in this VRML file. That is, in the example of FIG. 4, the whole moves in the X-axis direction according to the channel 2 while expanding and contracting.
[0047]
As shown in FIG. 4 to FIG. 6, the movement of each part can be additionally described in each “Separator” description section. For example, as shown in FIG. 5, it is described that the sphere of the first part (radius is 0.0136685) changes in size in the X-axis direction according to channel 3, that is, the sphere changes to an ellipsoid. Has been.
[0048]
By describing in the same manner, the movement can be described for each part separately from the whole. Therefore, by combining the movements of each part, it is possible to make while enjoying complex movements.
[0049]
In the first embodiment, the processed virtual object description data (VRML document shown in FIGS. 4 to 6) (the VRML document shown in FIGS. 4 to 6) is stored in the shape data storage unit 8 as it is. I was able to. For this reason, there is a problem that the copyright of the description data of the virtual object cannot be protected. On the other hand, it is possible to easily protect the copyright of the description data of the virtual object by storing only the moving image. In this case, a new label “Protect” is defined for informing the information processing device that copying of the description data of the virtual object itself is prohibited, and this is inserted at the beginning of the VRML document as shown in FIGS. To do. That is,

Is described. The animation control unit 4 of the information processing apparatus that processes the description data of the virtual object stores the processed result in the shape data storage unit 8 by inserting the label “Protect” at a predetermined position of the description data of the virtual object. When doing this, first, it is played back in the background and saved as simple moving image data (for example, an AVI file). In this way, the user can enjoy the processing of the virtual object and can protect the provider's right (for example, copyright) for the description data of the virtual object.
[0050]
As described above, according to the first embodiment, the user can perform choreography on a character such as a three-dimensional CG model while listening to the sound of the selected channel. In addition, animation can be created without knowledge of the CG model.
(Second Embodiment)
FIG. 7 schematically shows the configuration of an information processing apparatus according to the second embodiment of the present invention. In addition, the same code | symbol is attached | subjected to FIG. 1 and an identical part, and a different part is demonstrated. That is, in the information processing apparatus of FIG. 7, a motion input unit 10 is further added to the components shown in FIG.
[0051]
The motion input unit 10 mainly includes an image acquisition unit 10a and an image processing unit 10b, and the image processing unit 10b extracts the movement of the captured object from the image acquired by the image acquisition unit 10a.
[0052]
The motion information of the imaged object extracted by the motion input unit 10 is sent to, for example, the animation control unit 4 where the virtual object is moved based on the motion information. The motion element editing unit 5 creates a motion element based on the motion information and stores it in the motion element storage unit 6.
[0053]
In order to acquire the movement of the imaging target (for example, the movement of a human hand or the movement of the body) by the movement input unit 10, for example, using a CCD camera as the image acquisition unit 10 a, Alternatively, the body may be extracted from the background, and the motion may be extracted from the extracted image.
[0054]
The image acquisition unit 10a may receive a reflected light and acquire a distance image, for example, a distance image acquisition device as described in Japanese Patent Application No. 9-299648. Hereinafter, a case where the image acquisition unit 10a is a distance image acquisition device will be described as an example.
[0055]
Here, the image acquisition unit 10a (hereinafter referred to as the distance image acquisition unit 10a) that acquires the distance image and the distance image will be briefly described.
The appearance of the distance image acquisition unit 10a is shown in FIG. A light receiving unit 103 composed of a circular lens and an area sensor (not shown) at the rear thereof is arranged in the center, and the circular lens is surrounded by an LED that emits light such as infrared rays along its outline. A plurality of (for example, eight) light emitting units 101 are arranged at equal intervals.
[0056]
The light emitted from the light emitting unit 101 is reflected by the object, collected by the lens of the light receiving unit 103, and received by the area sensor at the rear of the lens. The area sensor is, for example, a sensor arranged in a 256 × 256 matrix, and the intensity of reflected light received by each sensor in the matrix becomes a pixel value. The image acquired in this way is a distance image as the intensity distribution of reflected light as shown in FIG.
[0057]
FIG. 9 shows a configuration example of the distance image acquisition unit 10a, which mainly includes a light emitting unit 102, a light receiving unit 103, a reflected light extraction unit 102, and a timing signal generation unit 104.
[0058]
The light emitting unit 101 emits light whose intensity varies with time in accordance with the timing signal generated by the timing signal generating unit 104. This light is applied to the target object in front of the light emitting unit.
[0059]
The light receiving unit 103 detects the amount of light reflected by the target object of the light emitted from the light emitting unit 101.
The reflected light extraction unit 102 extracts a spatial intensity distribution of the reflected light received by the light receiving unit 103. Since the spatial intensity distribution of the reflected light can be captured as an image, this is hereinafter referred to as a distance image.
[0060]
In general, the light receiving unit 103 simultaneously receives not only light reflected from an object of light emitted from the light emitting unit 101 but also external light such as illumination light and sunlight. Therefore, the reflected light extraction unit 102 takes the difference between the amount of light received when the light emitting unit 101 emits light and the amount of light received when the light emitting unit 101 does not emit light, thereby obtaining the light emitting unit 101. Only the reflected light component of the light from the target object is extracted.
[0061]
The reflected light extraction unit 102 extracts the intensity distribution, that is, a distance image as shown in FIG. 10, from the reflected light received by the light receiving unit 103.
For the sake of simplicity, FIG. 10 shows a case of an 8 × 8 pixel distance image that is a part of a 256 × 256 pixel distance image.
[0062]
The reflected light from the object decreases significantly as the distance of the object increases. When the surface of the object scatters light uniformly, the amount of light received per pixel in the distance image decreases in inverse proportion to the square of the distance to the object.
[0063]
In FIG. 10, the cell value (pixel value) in the matrix represents the intensity of the acquired reflected light in 256 gradations (8 bits). For example, a cell having a value of “255” is the state closest to the distance image acquisition unit 10a, a cell having a value of “0” is far from the distance image acquisition unit 10a, and reflected light is a distance image acquisition unit. 10a is not reached.
[0064]
Each pixel value of the distance image represents the amount of reflected light received by the unit light receiving unit corresponding to the pixel. Reflected light is affected by the nature of the object (specularly reflects, scatters, absorbs, etc.), the direction of the object, the distance of the object, etc., but the entire object scatters light uniformly. In this case, the amount of reflected light is closely related to the distance to the object. Since hands and the like have such properties, the distance image when the hand is put out in front of the distance image acquisition unit 1 reflects the distance to the hand, the inclination of the hand (the distance is partially different), and the like. A three-dimensional image as shown in FIG. 11 can be obtained.
[0065]
The image processing unit 10b of the motion input unit 10 has the format shown in FIG. 10 and performs edge extraction (imaging object contour extraction), centroid extraction, area calculation, and distance calculation to the imaging object. Various image processing for extracting information such as the shape, motion, and position of the image pickup body such as calculation of a motion vector is performed.
[0066]
In the image processing unit 10b, first, with respect to the distance image data as shown in FIG. 10, except for cells whose pixel values are equal to or less than a predetermined value, for example, the image is in the distance image as shown in FIG. The contour information of the imaged object is extracted.
[0067]
In order to extract the contour information as shown in FIG. 12, the pixel values of adjacent pixels are compared, a constant value is inserted only where the pixel value is equal to or greater than a certain value α, and consecutive images assigned with the same constant value. What is necessary is just to extract the pixel of an area | region.
[0068]
That is, if the pixel value at the coordinate position (i, j) on the matrix is P (i, j) and the pixel value of the contour information is R (i, j),

Thus, the contour information of the object as shown in FIG. 12 can be obtained.
[0069]
Based on the contour information as shown in FIG. 12 extracted from each distance image, the center of gravity of the object is calculated.
It can be determined whether or not the object is moving by comparing the position of the center of gravity with the position of the center of gravity extracted from the immediately preceding distance image. When it is determined that the object is moving, a motion vector representing the change amount and the change direction of the center of gravity is calculated. Further, not only the position of the center of gravity but also the contour information extracted from each of the continuous distance images may be compared with each other to obtain a motion vector of an arbitrary point on the contour.
[0070]
To calculate the distance d to the object from which the contour is extracted, first, a representative pixel value of an image in the vicinity of the center of gravity of the object is obtained. There are several representative pixel values such as an average value and a nearest neighbor value. Here, the nearest neighbor value is used. The intensity of reflected light from the object decreases in inverse proportion to the square of the distance to the object. That is, if the representative pixel value of the image of the object is Q (i, j),
Q (i, j) = K / d ² ... (1)
It can be expressed as.
[0071]
Here, for example, K is a coefficient adjusted so that the value of the pixel value R (i, j) becomes “255” when d = 0.5 m. The distance value d can be obtained by solving the equation (1) for d.
[0072]
In this way, information such as the shape, movement, and position of the target object can be extracted from each of the distance images.
FIG. 13 is a flowchart for explaining the processing operation of the information processing apparatus of FIG. The same parts as those in FIG. 2 are denoted by the same reference numerals, and different parts will be described. That is, the process of associating the motion with the virtual object in steps S6 to S7 in FIG. 2 is replaced with the process in steps S21 to S23 in FIG.
[0073]
When the virtual object is selected, the presentation unit 7 displays a motion editing window as shown in FIG. 14, for example.
In the motion editing window shown in FIG. 14, an area w2 for displaying the shape of the virtual object and a component selection menu bar w5 are displayed as in FIG. For example, if waist is selected, the motion editing window instructs the start of the distance image by the motion input window w14 and the distance image acquisition unit 10a as an interface for inputting the user's motion (for example, hand motion). A “motion input start” button B11 for performing the operation, a “motion input stop” button B12 for instructing the end of the distance image by the distance image acquisition unit 10a, and a music part (channel) extracted from the previously selected MIDI file ) Selection area w3.
[0074]
When the user selects the “motion input start” button B11 from the waiting state in step S3 (step S21), the distance image acquisition unit 10a starts acquiring the distance image of the hand of the user to be imaged (step S22). For example, when the user moves his / her hand within a predetermined imaging range of the distance image acquisition unit 10a as shown in FIG. 9, the distance image acquisition unit 10a captures and captures the movement at a rate of about 30 images per second, for example. The distance image is displayed on the motion input window w14. At the same time, as described above, the image processing unit 10b performs edge extraction (imaging object contour extraction), centroid extraction, area calculation, distance calculation to the imaging object, motion vector calculation, and the like on the acquired distance image. Image processing is performed to extract information such as the shape, movement, and position of the imaging body (user's hand). Based on the extracted motion information, the animation control unit 4 presents a drawing of the motion of the selected “waist” part according to the level of the sound of the channel “1” of the previously selected MIDI file. 7 (step S23).
[0075]
In the first embodiment described above, when a motion is added to the virtual object, it is based on a combination of detailed items displayed on the motion selection menu w4 of the motion editing window. In other words, the movement is one of “move left and right”, “move up and down” and “move back and forth”, the rotation is one of “rotate vertically”, “rotate horizontally”, and “rotate forward”. It was necessary to select and combine the movements one by one, such as “horizontal enlargement / reduction”, “vertical enlargement / reduction”, or “enlargement / reduction overall”. However, in 2nd Embodiment, it can input directly by a user's hand or body movement. For example, rotation and movement can be input at a time by moving the hand vertically while moving the hand vertically.
[0076]
The image processing unit 10b uses the movement vector of the gravity center position of the contour of the imaging target (for example, the hand) extracted from the distance image and the detailed items of the movement (for example, “move left and right”, “move up and down” “front and back” ”Move to”, “Rotate vertically”, “Rotate horizontally” or “Rotate forward” for rotation, “Scaling horizontally”, “Vertical scaling”, “Enlarging / reducing to the whole” ) Is stored in advance. Then, the movement of the imaging target may be detected by comparing the correspondence with the movement vector of the gravity center position of the contour of the imaging target extracted from the actually acquired distance image. That is, the motion of the imaging target may be determined based on, for example, the motion vector corresponding to the motion vector extracted from the distance image.
[0077]
Further, if the movement of the imaging target extracted from the distance image is analyzed in more detail (for example, a movement vector is obtained from an arbitrary point on the contour of the imaging target), the imaging target (for example, a hand) extracted from the distance image. A detailed movement corresponding to the movement of the virtual object can be associated with the virtual object.
[0078]
The input of the movement is terminated after a certain time, and the input state is entered again. At that time, since the image of one's hand or body is not displayed on the screen, the movement cannot be input.
[0079]
As described above, according to the second embodiment, the user can naturally input a motion using his / her hand or body, so that an animation can be created intuitively. In particular, it can be applied to music education using the body for infants.
(Third embodiment)
FIG. 15 schematically shows the configuration of an information processing apparatus according to the third embodiment of the present invention. In addition, the same code | symbol is attached | subjected to FIG. 1 and an identical part, and a different part is demonstrated. That is, in the information processing apparatus of FIG. 15, an evaluation unit 11 that evaluates the motion of the animation created by the user is added to the components shown in FIG.
[0080]
The evaluation unit 11 evaluates the movement and calculates the result by, for example, a score. For example, in accordance with the type of music such as enka, kayokyoku, word, waltz, weight values are determined in advance as wm, wr, wc for each movement (movement, rotation, enlargement / reduction), and each part of the virtual object ( For each i), the amount of movement associated with the user, that is, the movement amount mi, the rotation amount ri, and the deformation amount ci are calculated, and the evaluation value E is obtained from the following equation.
[0081]
[Expression 1]

[0082]
The evaluation value calculated by the evaluation unit 11 is presented to the presentation unit 7 or, if the evaluation value E exceeds, for example, a certain value, a message that praises “It's cool” or the like is presented. 7 may be presented. Furthermore, the information processing apparatus of the present invention may be applied to a game machine such as an arcade game so that a prize or the like is output when the evaluation value E exceeds a certain value.
[0083]
In addition to merely presenting the evaluation value to the user, it is also possible to compete for victory or defeat among a plurality of users by the evaluation value. Further, the evaluation format is not necessarily limited to the above method.
[0084]
【The invention's effect】
As described above, according to the present invention, it is possible to easily process and use information such as creation of animation by adding music or movement to a description of a virtual object in a three-dimensional or two-dimensional virtual space.
[Brief description of the drawings]
FIG. 1 is a diagram schematically showing a configuration of an information processing apparatus according to a first embodiment of the present invention.
FIG. 2 is a flowchart for explaining a processing operation of the information processing apparatus of FIG. 1;
FIG. 3 is a diagram showing an example of a motion editing window presented in a presentation unit.
FIG. 4 is a diagram showing an example of a VRML document in which a label description is inserted.
FIG. 5 is a diagram showing an example of a VRML document in which a label description is inserted.
FIG. 6 is a diagram showing an example of a VRML document in which a label description is inserted.
FIG. 7 is a diagram schematically showing a configuration of an information processing apparatus according to a second embodiment of the present invention.
FIG. 8 is a diagram showing an example of the appearance of a distance image acquisition unit.
FIG. 9 is a diagram illustrating a configuration example of a distance image acquisition unit.
FIG. 10 is a diagram illustrating an example of a distance image in which the intensity of reflected light is a pixel value.
11 is a diagram showing a three-dimensional representation of a distance image in a matrix format as shown in FIG.
FIG. 12 is a diagram showing an example of a contour image of an object extracted from a distance image.
13 is a flowchart for explaining a processing operation of the information processing apparatus in FIG. 7;
FIG. 14 is a diagram showing another example of a motion editing window presented in the presenting unit.
FIG. 15 is a diagram schematically showing a configuration of an information processing apparatus according to a third embodiment of the present invention.
[Explanation of symbols]
1 ... Music information input section
2 ... Music information extraction unit
3. Music information playback unit
4 ... Animation control unit
5 ... Movement element editor
6 ... Movement element storage unit
7 ... presentation part
8. Shape data storage unit
9 ... Input section
10 ... Movement input part
10a (distance) image acquisition unit
10b Image processing unit
11 ... Evaluation Department

Claims (6)

Each pixel value is a distance value 2 in the depth direction from the light emitting means for irradiating the imaging target with light, the light receiving means for receiving the reflected light from the imaging target, and the intensity distribution of the reflected light received by the light receiving means. An image acquisition unit including an image generation unit that generates a distance image that is an intensity of the reflected light that is inversely proportional to the power, a first storage unit that stores a description of each component of a virtual object including a plurality of components, and An information processing method in which a computer including a second storage unit that stores a description of a motion of a virtual object processes the virtual object,
A first step of extracting sound components from input sound data;
A second step of selecting a desired part from the parts of the virtual object;
A third step of acquiring, in the image acquisition device, a plurality of time-series imaged distance images of an imaging target that moves to input a movement of a selected component;
A fourth step of recognizing movement of the imaging target from contour information of the imaging target extracted from each of the plurality of distance images;
The description of the selected part includes a description of the action selected from the description of the action stored in the second storage means and a description of the action corresponding to the movement recognized in the fourth step. And a fifth step of inserting a label corresponding to each of the sound components selected from the sound components extracted by the extraction means,
Based on the operation description and sound component corresponding to each label inserted in the description of the part of the virtual object, the operation described in the operation description according to the sound component A sixth step of presenting a virtual object to perform
An information processing method comprising: Each pixel value is a distance value 2 in the depth direction from the light emitting means for irradiating the imaging target with light, the light receiving means for receiving the reflected light from the imaging target, and the intensity distribution of the reflected light received by the light receiving means. An image acquisition device including an image generation unit that generates a distance image that is an intensity of the reflected light that is inversely proportional to the power, and a storage unit that stores a description of each part of a virtual object composed of a plurality of parts. An information processing method for processing a virtual object,
A first step of extracting sound components from input sound data;
A second step of selecting a desired part from the parts of the virtual object;
A third step of acquiring, in the image acquisition device, a plurality of time-series imaged distance images of an imaging target that moves to input a movement of a selected component;
A fourth step of recognizing movement of the imaging target from contour information of the imaging target extracted from each of the plurality of distance images;
In the description of the selected part, the description of the action corresponding to the movement recognized in the fourth step and the sound component selected from the sound components extracted in the first step A fifth step of inserting corresponding labels into each;
Based on the operation description and sound component corresponding to each label inserted in the description of the part of the virtual object, the operation described in the operation description according to the sound component A sixth step of presenting a virtual object to perform
An information processing method comprising: First storage means for storing a description of each part of a virtual object composed of a plurality of parts;
Second storage means for storing a description of the motion of the virtual object;
Extraction means for extracting sound components from the input sound data;
First selecting means for selecting a desired part from among the parts of the virtual object;
A light emitting means for irradiating light to an imaging target that moves to input the movement of the selected component;
A light receiving means for receiving reflected light from the imaging target;
Image generation means for generating a distance image having an intensity of the reflected light in which each pixel value is inversely proportional to the square of the distance value in the depth direction from the intensity distribution of the reflected light received by the light receiving means;
Motion recognition means for recognizing movement of the imaging target from contour information of the imaging target extracted from each of a plurality of time-series distance images generated by the image generation means;
One of the description of the action selected from the description of the action stored in the second storage means and the description of the action corresponding to the movement recognized by the recognition means in the description of the selected part And insertion means for inserting labels corresponding to each of the sound components selected from the sound components extracted by the extraction means;
Based on the operation description and sound component corresponding to each label inserted in the description of the part of the virtual object, the operation described in the operation description according to the sound component Presenting means for presenting a virtual object to perform,
An information processing apparatus comprising: First storage means for storing a description of each part of a virtual object composed of a plurality of parts;
Extraction means for extracting sound components from the input sound data;
First selecting means for selecting a desired part from among the parts of the virtual object;
A light emitting means for irradiating light to an imaging target that moves to input the movement of the selected component;
A light receiving means for receiving reflected light from the imaging target;
Image generation means for generating a distance image having an intensity of the reflected light in which each pixel value is inversely proportional to the square of the distance value in the depth direction from the intensity distribution of the reflected light received by the light receiving means;
Motion recognition means for recognizing movement of the imaging target from contour information of the imaging target extracted from each of a plurality of time-series distance images generated by the image generation means;
The description of the selected part corresponds to the description of the action corresponding to the movement recognized by the movement recognition means and the sound component selected from the sound components extracted by the extraction means. Inserting means for inserting a label to be
Based on the operation description and sound component corresponding to each label inserted in the description of the part of the virtual object, the operation described in the operation description according to the sound component Presenting means for presenting a virtual object to perform,
An information processing apparatus comprising: Each pixel value is a distance value 2 in the depth direction from the light emitting means for irradiating the imaging target with light, the light receiving means for receiving the reflected light from the imaging target, and the intensity distribution of the reflected light received by the light receiving means. An image acquisition unit including an image generation unit that generates a distance image that is an intensity of the reflected light that is inversely proportional to the power, a first storage unit that stores a description of each component of a virtual object including a plurality of components, and A computer comprising second storage means for storing a description of the motion of the virtual object;
Extraction means for extracting sound components from the input sound data;
Selecting means for selecting a desired part from the parts of the virtual object;
Acquisition means for acquiring, in the image acquisition device, a plurality of time-series distance images captured of an imaging target that moves to input the movement of a selected component;
Recognizing means for recognizing movement of the imaging target from contour information of the imaging target extracted from each of the plurality of distance images;
One of the description of the action selected from the description of the action stored in the second storage means and the description of the action corresponding to the movement recognized by the recognition means in the description of the selected part On the other hand, insertion means for inserting labels corresponding to each of the sound components selected from the sound components extracted by the extraction means,
Based on the operation description and sound component corresponding to each label inserted in the description of the part of the virtual object, the operation described in the operation description according to the sound component A machine-readable recording medium on which a program for functioning as a presentation unit for presenting a virtual object is recorded. Each pixel value is a distance value 2 in the depth direction from the light emitting means for irradiating the imaging target with light, the light receiving means for receiving the reflected light from the imaging target, and the intensity distribution of the reflected light received by the light receiving means. An image acquisition device including an image generation unit that generates a distance image that is an intensity of the reflected light that is inversely proportional to the power, and a computer including a storage unit that stores a description of each part of a virtual object composed of a plurality of parts.
Extraction means for extracting sound components from the input sound data;
First selection means for selecting a desired part from among the parts of the virtual object;
Acquisition means for acquiring, in the image acquisition device, a plurality of time-series distance images captured of an imaging target that moves to input the movement of a selected component;
Motion recognition means for recognizing movement of the imaging target from contour information of the imaging target extracted from each of a plurality of distance images acquired by the acquisition means;
The description of the selected part corresponds to the description of the action corresponding to the movement recognized by the movement recognition means and the sound component selected from the sound components extracted by the extraction means. Insertion means for inserting a label to be
Based on the operation description and sound component corresponding to each label inserted in the description of the part of the virtual object, the operation described in the operation description according to the sound component A machine-readable recording medium having recorded thereon a program that functions as a presenting means for presenting a virtual object.

Images