JP4185437B2 - Video composition output device and audio reproduction output device - Google Patents

Description

  The present invention relates to a video composition output device and an audio reproduction output device, and in particular, a card on which characters or patterns for identifying a card are drawn, and a moving image corresponding to the card information and movement, The present invention relates to a video composition output device and an audio reproduction output device for outputting original motion video or audio with high accuracy.

  Conventionally, an apparatus for photographing a card with a pattern and outputting a MIDI (Musical Instrument Digital Interface) signal for controlling a musical instrument based on the type of card and three-dimensional position information detected from the photographed camera image, and A technique regarding an apparatus that outputs video is known (for example, see Patent Document 1).

  In Patent Document 1, three-dimensional information including a three-dimensional position and posture of a card is detected based on a video stream signal obtained by photographing a card held by an operator, and MIDI data corresponding to the three-dimensional information is output. . In Patent Document 1, an image corresponding to three-dimensional information (coordinate position, rotation angle, tilt angle, etc.) obtained by photographing a card is displayed.

In addition, 3D shape data can be considered as an image to be displayed, but as a technique for acquiring 3D shape data necessary for display, for example, the approximate shape of a subject is measured and the approximate shape is obtained. There is a technique that can obtain a more detailed shape of a subject based on block matching and further obtain distance information from the camera (see, for example, Patent Document 2).
JP 2002-32076 A Japanese Patent Laid-Open No. 2003-271928

  However, Patent Document 1 only outputs audio and video corresponding to each card. For example, video and audio output corresponding to the moving speed of a card photographed by a camera is not considered. In addition, when a plurality of cards are photographed at the same time, video and audio corresponding to the combination and positional relationship cannot be output. For this reason, it cannot be said that high-accuracy video and audio output using a card is realized.

  The present invention has been made in view of the above-described problems, and based on card information and card speed information detected from an image obtained by photographing a card, a moving image and a three-dimensional moving image are actually photographed. A video synthesis output device that synthesizes a camera video and outputs a high-accuracy video, and a video synthesis output device and an audio playback output device for realizing high-accuracy audio output based on card information and card speed information The purpose is to provide.

  In order to solve the above problems, the present invention employs means for solving the problems having the following characteristics.

  The invention described in claim 1 synthesizes another video signal to the video signal based on the card information including the card type, the three-dimensional position information, and the posture information based on the video signal of the camera that shoots the card. A speed detection unit that detects a card speed based on the card information obtained for each frame at a predetermined frame interval of a video signal shot by the camera, and the card And a video output unit that selects and outputs a video signal to be synthesized based on the information and the speed information obtained by the speed detection unit.

  According to the first aspect of the present invention, various videos corresponding to the movement contents of the card can be output by selecting a video using not only the card information but also the speed information. As a result, it is possible to output a highly accurate video with excellent interactivity.

  According to a second aspect of the present invention, the video output unit selects a moving image based on the card information and the speed information obtained by the speed detection unit, and the three-dimensional position information, the posture information, Based on the above, the display, non-display, size, position, and direction of the moving image are controlled and output.

  According to the second aspect of the present invention, it is possible to realize a higher-accuracy video output by selecting a video image based on the card information and the speed information, and further controlling and outputting the size, the display position, and the like. it can.

  According to a third aspect of the present invention, the video output unit is a third order based on the card information, the speed information obtained by the speed detection unit, and the connection information obtained from the color information and the three-dimensional position information. An original moving image is selected, and the display, non-display, size, position, and direction of the three-dimensional moving image are controlled and output based on the three-dimensional position information and the posture information.

  According to the invention described in claim 3, more accurate video can be obtained by selecting a three-dimensional video based on card information, speed information, and connection information, and further controlling the size, display position, etc. and outputting the video. Output can be realized.

  According to a fourth aspect of the present invention, when there are a plurality of cards in a video signal photographed by the camera, the speed detection unit detects relative speed information between the plurality of cards based on the card information. It is characterized by doing.

  According to the fourth aspect of the present invention, it is possible to output various videos corresponding to the movement contents between the cards by selecting the video using the card information and the relative speed information for each of the plurality of cards. As a result, it is possible to output a highly accurate video with excellent interactivity.

  The invention described in claim 5 includes a mutual position / posture detection unit that detects mutual position / posture information including the distance between the cards and the posture information from the card information of the plurality of cards.

  According to the fifth aspect of the present invention, it is possible to output various images corresponding to movement between cards and content of posture using the mutual position and posture information.

  According to a sixth aspect of the present invention, the video output unit selects and outputs the moving image and / or the three-dimensional moving image based on the relative speed information and the mutual position and orientation information. Features.

  According to the sixth aspect of the invention, by using the relative speed information and the mutual position / posture information, it is possible to realize more accurate video output.

According to the seventh aspect of the present invention, the sound reproduction for reproducing and outputting the sound accumulated in advance based on the card information including the card type, the three-dimensional position information, and the posture information by the video signal of the camera for photographing the card. In the output device, based on the card information obtained for each frame at a predetermined frame interval of the video signal photographed by the camera, a speed detection unit that detects a card speed, and the card information and the speed detection unit based on the obtained velocity information, if you select the audio file possess an audio reproduction output unit for reproducing output, a plurality of card is present in the video signal captured by the camera, the speed detection unit, Relative speed information between the plurality of cards is detected based on the card information .

According to the seventh aspect of the invention, it is possible to output various sounds corresponding to the movement contents of the card by selecting the sound using not only the card information but also the speed information. In addition, by selecting sound using card information and relative speed information for each of a plurality of cards, various sounds corresponding to the movement contents between the cards can be output. This makes it possible to output highly accurate voice with excellent interactivity.

The invention described in claim 8 includes a mutual position / posture detecting unit that detects mutual position / posture information including the distance between the cards and the posture information from the card information of the plurality of cards.

According to the eighth aspect of the invention, it is possible to output various sounds corresponding to movement between cards and posture contents using the mutual position and posture information.

The invention described in claim 9 is characterized in that the audio reproduction output unit selects and outputs the audio file based on the relative velocity information and the mutual position and orientation information.

According to the ninth aspect of the invention, by using the relative velocity information and the mutual position and orientation information, it is possible to realize more accurate voice output.

  ADVANTAGE OF THE INVENTION According to this invention, based on the card information obtained from the camera image | video which image | photographed the card | curd, and the speed information of a card | curd, the highly accurate image | video or audio | voice excellent in the interactive property can be output.

<Outline of the present invention>
The present invention shoots a card on which identification information such as a specific character or pattern is drawn, and detects card information including card type, three-dimensional position information, and posture information detected from the captured camera image, and card speed information. Based on the above, the corresponding video is selected from the previously stored moving image and 3D video, and is combined with the actually captured camera video and output.

  Here, the three-dimensional position information represents position information on the x, y, and z coordinates of the card obtained by photographing with the camera. The above contents will be described with reference to the drawings. FIG. 1 is a diagram illustrating an example of a positional relationship between a card and a camera. Here, characters (a in FIG. 1) are drawn on the card 11 in order to specify the type (type) of the card. As long as the card can be identified, it is not limited to letters, but may be numbers, patterns, facial photographs, and the like.

  As shown in FIG. 1, in the position information of the card 11, the x coordinate indicates a position in the horizontal direction with respect to the camera, the y coordinate indicates a position in the vertical direction with respect to the camera, and the z coordinate indicates a distance with respect to the camera. Yes. The posture information indicates the degree of rotation based on the position of a predetermined card with respect to the above-described x-axis, y-axis, and z-axis, and represents the posture of the card itself. Thus, when the user moves the card, card information (card type, three-dimensional position information, posture information) is acquired from the camera that has photographed the card.

  In the present invention, the difference between the three-dimensional position information and the posture information is acquired from a plurality of frames detected from the camera image at a preset frame interval, and the moving speed of the card is detected from the acquired difference. A corresponding moving image and / or 3D moving image is selected based on the detected speed information, and is combined with a video signal of a camera that is actually shot and output.

  Specifically, card information consisting of card type, three-dimensional position information, and posture information is obtained by performing image processing such as area extraction and pattern matching of cards on which identification information such as specific characters and patterns are drawn. get. Various image processing techniques for acquiring the above card information already exist. For example, “Image Processing Engineering” by Ryoichi Suematsu and Hirohisa Yamada, Corona (hereinafter referred to as Non-Patent Document 1). ), "Augmented reality system based on marker tracking and its calibration" Hirokazu Kato, Mark Billinghurst, Koichi Asano, Keihachi Tachibana, basic paper TVRSJ vol4 No. 4 1999 (hereinafter, referred to as non-patent document 2) can be applied.

  Also, based on the card information, a video image stored in advance or inputted from the outside and a 3D moving image stored by connection information of vertex groups having color information and 3D coordinate information are projected. Conversion, viewpoint conversion, and output, and using a video synthesis method such as chroma key processing, the video is synthesized with the camera video actually shooting the card and output.

  Here, projection transformation and viewpoint transformation are three-dimensional as described in, for example, “OpenGL Programming Guide” by Jackie Neider, Tom Davis, Mason Woo, Addison Wesley (hereinafter referred to as Non-Patent Document 3). Projective transformation and viewpoint transformation techniques using OpenGL, which is a format for expressing a space, can be applied. As for image synthesis technology, for example, “Practical image processing learned in C language” (Masayuki Inoue, Nobuyuki Yagi, Masaki Hayashi, Eisuke Nakasu, Koji Mitani, Masato Okui, Ohmsha) Video composition such as chroma key processing described in the above.

Furthermore, the above-described color information is, for example, when the card area is formed in a rectangle, each color information (R ₁ , G ₄ ) of the vertices (P ₁ , P ₂ , P ₃ , P ₄ ) of the rectangle. ₁ , B ₁ ) to (R ₄ , G ₄ , B ₄ ), and the above-described connection information is formed from, for example, the above-described rectangular vertices (P ₁ , P ₂ , P ₃ , P ₄ ) 2 The point connection information in a region formed as two triangles (P ₁ , P ₂ , P ₄ ) and (P ₁ , P ₃ , P ₄ ) is shown. It should be noted that the colors in the triangular area are set by interpolating the color information of the triangles (P ₁ , P ₂ , P ₄ ) and (P ₁ , P ₃ , P ₄ ) described above, It corresponds to the surface of the shape. As a technique for forming a three-dimensional moving image using color information in this manner, for example, “Note VRML2.0 Reference Manual” (Rick Carey, Gavin Bell, Addison Wesley) (hereinafter referred to as Non-Patent Document 5). There is.

  Further, in the present invention, based on the card type photographed by the camera, the three-dimensional position information, the card information consisting of the posture information and the card speed information, the type of the audio file reproduced and output from the pre-stored audio file, Select volume and panning, and play and output the audio file. Here, the audio file format includes wav, aiff, MP3, WMA, VQ, rm, ram, mov, aac, atrac3, and the like.

  Furthermore, in the present invention, when a plurality of cards are photographed, from the mutual type, the three-dimensional position information, and the posture information among the plurality of cards, the moving image, the display of the three-dimensional moving image, the non-display, the size, Controls the position, direction, and on / off, volume, panning, and type of audio output of the audio file.

<Example 1: Video composition output device>
Next, a video composition output apparatus for synthesizing and outputting a moving image and a three-dimensional moving image to which the present invention is applied with a video signal from a camera that is actually photographed will be described with reference to the drawings.

  FIG. 2 is a diagram showing a configuration example of a video composition output system according to the present invention. The video composition output system of FIG. 2 is configured to include a camera 22 for photographing the card 21, a card information acquisition device 23, and a video composition output device 24. In addition, the video composition output device 24 is configured to include a speed detection unit 25, a moving image output unit 26, a 3D moving image output unit 27, and a video composition output unit 28.

  In the video composition output system shown in FIG. 2, first, the camera 22 photographs a card 21 on which a specific character or pattern for identifying the card is drawn. The camera video signal photographed by the camera 22 is output to the card information acquisition device 23 and the video composition output unit 28. For example, the card 21 is held by a user and moves or rotates with respect to the camera 21 in three-dimensional coordinates (x, y, z coordinates shown in FIG. 1).

  The card information acquisition device 23 extracts the area of the card 21 from the camera video signal input from the camera 22, performs image processing such as pattern matching, and the type of card included in the captured video, and the three-dimensional position information. The card information consisting of posture information is acquired. Further, the card information acquisition device 23 outputs the acquired card information to the speed detection unit 25, the moving image output unit 26, and the 3D moving image output unit 27. When the card information acquisition device 23 extracts the area of the card 21, for example, as shown in Non-Patent Document 2, a rectangle is drawn in the card 21 of FIG. By acquiring the coordinates, it is possible to acquire three-dimensional position information and posture information.

  The speed detection unit 25 determines the three-dimensional position at a preset frame interval included in the video signal for each card type from the card type, the three-dimensional position information, and the posture information sequentially input from the card information acquisition device 23. Information and posture information is obtained, the difference between the three-dimensional position information and posture information is calculated to determine the moving speed including the direction of each card, and the card type and speed information is obtained from the video image output unit 26 and the three-dimensional motion information. The video is output to the video output unit 27. Here, for example, if the video signal has 30 frames per second, the frame interval is set to 15 frames.

  The moving image output unit 26 selects a corresponding moving image from moving image data stored in advance or moving image input from the outside based on the type and moving speed of the input card, and further, the three-dimensional card Based on the position information and the posture information, the moving image to be output is moved and rotated by projection conversion and viewpoint conversion, and is output to the image composition output unit 28 as a moving image signal. Note that the video output by the projective conversion and the viewpoint conversion is output on the card actually captured by the camera 22, and the video whose direction is adjusted in accordance with the direction of the card is output. Thereby, it is possible to display by changing the direction of the video according to the direction of the card.

  Note that the position of the video to be output is not limited to this in the present invention. For example, the video can be set to be output at a position away from the card by a predetermined position.

  The 3D moving image output unit 27 includes speed information input by the speed detecting unit 25 and the type of card input by the card information acquisition device 23 from the previously stored 3D moving image data. A corresponding three-dimensional moving image is selected from the card information including three-dimensional position information and posture information, and the above-described vertex group connection information having color information and three-dimensional coordinate information.

  Further, based on the three-dimensional position information and posture information of the card 21, the output three-dimensional moving image is projected and converted by the viewpoint, moved and rotated, and drawn by a generally available three-dimensional moving image drawing software such as OpenGL. Then, a three-dimensional moving image is generated, and the generated three-dimensional moving image signal is output to the image composition output unit 28.

  As a three-dimensional space format for expressing the three-dimensional space (three-dimensional virtual reality space), a three-dimensional space format such as VRML, DirectX, DXF, etc. can be applied in addition to OpenGL.

  Here, an example of data for selecting videos stored in the moving image output unit 26 and the 3D moving image output unit 27 will be described with reference to the drawings.

  FIG. 3 is a diagram illustrating an example of video selection data stored in each of the moving image output unit and the 3D moving image output unit.

  The data items shown in FIG. 3 include a card type that specifies the type of card, a speed condition detected by the speed detection unit 25, a video corresponding to the card type and the speed condition, and the like. Further, the data stored in the 3D moving image output unit 27 includes the above connection information.

  In FIG. 3, for example, when the card type is a and the moving speed of the card measured at a predetermined frame interval is 7 cm / s, an image in which the person A is walking is selected. Also, when the card type is a and the moving speed of the card measured at a predetermined frame interval is 15 cm / s, an image in which the person A is running is selected. As described above, the selected video differs depending on the moving speed of the card. Based on the selected video, each of the video video output unit 26 and the 3D video output unit 27 selects a video and performs projection conversion and viewpoint conversion based on the 3D coordinate information and orientation information included in the card information. I do. Further, the video is displayed, hidden, controlled in size, position, direction, etc., and output to the video composition output unit 28.

  In the video composition output unit 28, the video of the actually captured camera input from the camera 22, the moving image input from the moving image output unit 26, and the 3D input from the 3D moving image output unit 27. The moving image signal is synthesized by image synthesizing software using key signal processing or the like, and a synthesized signal of the camera image, the moving image image, and the three-dimensional moving image is output as a video signal by a display device, a projector, or the like. The specific contents to be output will be described later.

  Here, with regard to the key signal processing, for example, a video can be synthesized by using chroma key signal processing or the like as shown in Non-Patent Document 4 described above.

  Further, in the video composition output unit 28, if a video signal is input from only one of the moving image output unit 26 or the 3D moving image output unit 27, the video signal obtained from one of the video signals is input from the camera 22. The video signal is synthesized and output.

  As described above, by using the speed information obtained by the speed detector 25, it is possible to select and display an optimal video. Thereby, it is possible to output a highly accurate video with excellent interactivity with respect to the card moved by the user.

<Example 2: Audio reproduction output device>
Next, the configuration of an audio reproduction output system to which the present invention is applied will be described with reference to the drawings.

  FIG. 4 is a diagram showing an example of the configuration of an audio reproduction output system according to the present invention. The audio output system shown in FIG. 4 is configured to include a camera 42 that captures a card 41, a card information acquisition device 43, and an audio reproduction output device 44. The audio reproduction output device 44 is configured to include a speed detection unit 45 and an audio reproduction output unit 46. Here, the card 41 is the same as the card 21 described above.

  First, the camera 41 photographs a card 41 on which identification information such as characters and patterns for specifying the card is drawn. A camera video signal photographed by the camera 42 is output to the card information acquisition device 43. Note that the card 41 is held by the user and moves or rotates with respect to the camera 42 in three-dimensional coordinates (x, y, z coordinates shown in FIG. 1).

  The card information acquisition device 43 extracts the area of the card 41 from the camera video signal input from the camera 42, performs the image processing such as pattern matching, and the type of the card included in the video and 3D position information. The card information consisting of posture information is acquired. Further, the card information acquisition device 43 outputs the acquired card information to the speed detection unit 45 and the audio reproduction output unit 46.

  The speed detection unit 45 determines the three-dimensional position at a preset frame interval included in the video signal for each card type from the card type, three-dimensional position information, and posture information sequentially input from the card information acquisition device 43. The information and posture information are acquired, the difference between the three-dimensional position information and posture information is calculated to determine the moving speed including the direction for each card, and the card type and speed information is output to the audio reproduction output unit 46.

  The audio reproduction output unit 46 selects an audio file stored in advance from an input card type and moving speed or an audio file input from the outside, reproduces an audio signal, and outputs a speaker. To a voice output device.

  In addition, in the sound output, for example, when surround playback output using a plurality of speakers is possible, the sound is output to a predetermined speaker based on the speed information, the three-dimensional position information, and the posture information. In addition, the sound volume is changed according to the distance from the card and the camera, and sound control such as panning, effects, and reverberation is performed. This makes it possible to output a realistic sound.

  Here, an example of data for selecting the voice of the voice reproduction output unit 46 will be described with reference to the drawings. FIG. 5 is a diagram illustrating an example of data included in the audio reproduction output unit. The data items shown in FIG. 5 are for outputting sound from the card type obtained from the card type, the speed condition obtained from the speed detection unit 45, and the card type and speed condition.

  In FIG. 5, for example, if the card type is b and the speed is 2 cm / s, an audio file whistling in person B's voice is selected. As an audio file, an audio file formatted in wav, aiff, MP3, WMA, VQ, rm, ram, mov, aac, atrac3, etc. can be used.

  As described above, the audio reproduction output device to which the present invention is applied can output a highly accurate audio file with excellent interactivity to the card moved by the user based on the speed information.

  Here, the above-described video composition output device 24 and audio reproduction output device 44 may be combined. Thereby, video output and audio output with higher accuracy can be realized.

  In addition, the card photographed by the camera may be not only one but also a plurality of cards. In the case of a plurality of images, it is possible to perform the same operation as when each image is photographed as shown in the above-described embodiment. By selecting the video and audio to be output based on the speed and the mutual position, it is possible to realize more accurate video output and audio output.

  Here, a video composition output device and an audio reproduction output device to which the present invention is applied when a plurality of cards are photographed as described above will be described. In the third embodiment to be described later, an example in which video synthesis output and audio reproduction output are realized by one device configuration will be described. However, as shown in the above-described embodiments, each device has a separate device configuration. You can also.

<Example 3: Video synthesis / audio reproduction output device>
FIG. 6 is a diagram showing a configuration example of a video synthesis / audio reproduction output system according to the present invention. The video synthesis / audio reproduction output system shown in FIG. 6 is configured to include a camera 62 for photographing the cards 61-1 and 61-2, a card information acquisition device 63, and a video synthesis / audio reproduction output device 64. Yes.

  The video synthesis / audio reproduction output device 64 includes a speed detection unit 65, a mutual position / posture detection unit 66, a moving image output unit 67, a 3D moving image output unit 68, an audio reproduction output unit 69, and an image. And a composite output unit 70.

  Here, in the embodiment shown in FIG. 6, an example in which two cards 61-1 and 61-2 are used will be described. However, the present invention is not limited to this, and three or more cards may be used.

  First, the cards 61-1 and 61-2 are photographed by the camera 62. The camera video signal photographed by the camera 62 is output to the card information acquisition device 63 and the video composition output unit 70. The cards 61-1 and 61-2 are held by the user and move or rotate with respect to the camera 62 in three-dimensional coordinates (x, y, and z coordinates shown in FIG. 1).

  The card information acquisition device 63 extracts the respective areas of the cards 61-1 and 61-2 from the camera video signal input from the camera 62 and performs image processing such as pattern matching to be included in the video imaged. The card information including the card type, three-dimensional position information, and posture information is acquired. Further, the card information acquisition device 63 outputs the acquired card information to the speed detection unit 65 and the mutual position / posture detection unit 66.

  The speed detection unit 65 is configured to detect the card type, the three-dimensional position information, and the posture information that are sequentially input from the card information acquisition device 63 at a preset frame interval included in the video signal for each of the plurality of card types. Obtain 3D position information and posture information, calculate the difference between the 3D position information and posture information, find the relative movement speed including the direction between the cards, type of card, between multiple cards taken The relative speed information is output to the moving image output unit 67, the 3D moving image output unit 68, and the audio reproduction output unit 69.

  Further, the mutual position / posture detection unit 66 detects mutual position / posture information (distance and posture information between cards) indicating at which position and in which direction the cards face each other from the card information of each input card. Then, the card information and the mutual position / posture information are output to the moving image output unit 67, the 3D moving image output unit 68, and the audio reproduction output unit 69.

  In the moving image output unit 67, a corresponding moving image from moving image data stored in advance or moving image input from the outside is obtained from the input card information, the relative speed information between the cards, and the mutual position and orientation information. Further, based on the three-dimensional position information and posture information of the cards 61-1 and 61-2, the moving image to be output is moved and rotated by projective transformation and viewpoint transformation, and the video composition output unit 70 is converted into a moving image signal. Output to. Note that the video output by the projection conversion and the viewpoint conversion is output on the card actually captured by the camera 62, and further, the video whose direction is adjusted according to the direction of the card is output.

  Also, the 3D moving image output unit 68 includes, among prestored 3D moving image data, relative speed information and mutual position / posture information, card information, and connection between cards input by the speed detecting unit 65. Select the corresponding 3D video from the information. Further, based on the three-dimensional position information and posture information of the cards 61-1 and 61-2, the three-dimensional moving image to be output is projected and converted, and the viewpoint is converted to move and rotate to generally use three-dimensional such as OpenGL. Drawing is performed by the moving picture drawing software to generate a three-dimensional moving picture, and the generated three-dimensional moving picture signal is output to the video composition output unit 70.

  The video composition output unit 70 is a camera image actually input from the camera 62, a moving image input from the moving image output unit 67, and a 3D input from the 3D moving image output unit 68. A moving image signal is synthesized by key signal processing or the like, and a synthesized signal of a camera image, a moving image, and a three-dimensional moving image is output as a video signal by a display device, a projector, or the like. The specific contents to be output will be described later.

  On the other hand, in the audio reproduction output unit 69, a corresponding audio file is selected from audio files stored in advance from input card information, relative speed information between cards, and mutual position and orientation information, or audio input from the outside. Select, reproduce the audio signal, and output it to an audio output device such as a speaker.

  For example, when surround playback output using a plurality of speakers is possible in audio output, audio is output to a predetermined speaker based on relative speed information and mutual position and orientation information. This makes it possible to output a realistic sound.

  Here, an example of data for selecting videos stored in the moving image output unit 67, the three-dimensional moving image output unit 68, and the audio reproduction output unit 69, and an image output to the display based on the data. Specific examples of output and audio output will be described with reference to FIGS.

  FIG. 7 is a diagram illustrating an example of data for selecting videos stored in the moving image output unit, the 3D moving image output unit, and the audio reproduction output unit. FIG. 8 is a diagram showing an example of video output and audio output based on the data example of FIG.

Here, FIG. 7A is an example of data provided in the moving image output unit 67 and the 3D moving image output unit 68, and FIG. 7B is an example of data provided in the audio reproduction output unit 69. In FIG. 7, information on the relative speed corresponding to at least two card types (card types), video of the card at the mutual position (FIG. 7A), or sound (FIG. 7B) is accumulated. . In the case of the 3D moving image output unit 68, connection information is also included in addition to the above items.
The data in FIG. 7 shows an example of data when two cards are photographed. However, in the present invention, the number of cards is not limited to this, and data is stored corresponding to a plurality of cards. You can also. When there is one card, the data shown in FIG. 3 or FIG. 5 is referred to. Furthermore, when there are a plurality of videos corresponding to the condition in the data, all of them may be selected, or a plurality of videos and sounds can be selected with priority.

  For example, as shown in FIG. 8A, there are two cards 81-1 and 81-2 that are photographed at the same time, and the type (card type) of the card 81-1 is a, The type (card type) is b. Here, output of video and audio when the card 81-1 and the card 81-2 are brought close to each other will be described.

  First, when the relative movement speed of the two cards obtained by the speed detection unit 65 is 0.5 cm / s and the mutual position obtained from the mutual position / posture detection unit 66 is at a position of 2 cm, from FIG. The video that person A bows is selected as video 1 corresponding to the card type a, and the video output by the video video output unit 67 and the 3D video output unit 68 is combined with the video actually captured. The video is output at a predetermined position (FIG. 8B). Also, from FIG. 7B, an audio file to be output as “Konichiha” is selected as the corresponding audio, and is reproduced and output from an audio output device such as a speaker.

  On the other hand, as the video 2 corresponding to the card type b, the video that the person B bows from the data in FIG. 7A is selected, and the video output from the moving image output unit 67 and the 3D moving image output unit 68 is actually displayed. Is combined with the video imaged in (1) and output to a predetermined position (FIG. 8B). Also, from FIG. 7B, an audio file to be output as “Domo” is selected and reproduced and output from an audio output device such as a speaker.

  Further, when the relative movement speed of the two cards obtained by the speed detection unit 65 is 0.5 cm / s and the mutual position obtained from the mutual position / posture detection unit 66 is 0.2 cm, FIG. ) Is selected as the video 1 corresponding to the card type a, and the video output by the moving image output unit 67 and the 3D moving image output unit 68 is actually shot. For example, the video is output by performing control such as displaying on the card 81-2 (FIG. 8C). Also, from FIG. 7B, an audio file that outputs “Yoroshiku” as the corresponding audio is selected and reproduced and output from an audio output device such as a speaker.

  On the other hand, the video 2 corresponding to the card type b is set not to display the video output from the data of FIG. That is, in the video of the corresponding card type a, the video in which the person A and the person B are shaking hands is displayed, so that the card type b does not need to display the video. By displaying in this way, it is possible to display an image such as a person embracing each other or a person getting into a car with a more realistic operation, and a more detailed image can be displayed with high accuracy.

  Also, from FIG. 7 (b), a voice file that outputs “Yoroshiku” as a corresponding voice is selected and output from a voice output device such as a speaker.

  Furthermore, when the relative movement speed of the two cards obtained by the speed detection unit 65 is 15 cm / s and the mutual position obtained from the mutual position / posture detection unit 66 is at a position of 0.5 cm, from FIG. A video in which the person A falls is selected as the video 1 corresponding to the card type a, and the video output by the video video output unit 67 and the three-dimensional video output unit 68 is combined with the video actually captured, An image is output at a predetermined position (FIG. 8D). Also, from FIG. 7B, an audio file to be output as “Wor” as the corresponding audio is selected and output from an audio output device such as a speaker.

  On the other hand, as the video 2 corresponding to the card type b, the video in which the person B falls is selected from the data of FIG. 7A, and the video output by the moving image output unit 67 and the 3D moving image output unit 68 is actually used. The video is synthesized with the video being shot and output to a predetermined position (FIG. 8D). Also, from FIG. 7B, an audio file to be output as “Itai” is selected and output from an audio output device such as a speaker.

  Thus, by selecting video and audio based on the relative velocity information and the mutual position and orientation information, it is possible to output video and audio that are more diverse and highly accurate than when one card is used. This makes it possible to output highly accurate video and audio with excellent interactivity with respect to the card moved by the user.

  As described above, according to the present invention, a video image, a three-dimensional video image is obtained based on card information obtained from a video obtained by photographing a card on which identification information such as a specific character or pattern is drawn, and speed information of the card. By selecting the audio and synthesizing it with the actually captured video signal, video output and audio output can be realized with high accuracy. This makes it possible to output highly accurate video and audio with excellent interactivity with respect to the card moved by the user.

  In addition, when there are multiple cards in the video signal that was actually shot, actual video was shot by selecting video, 3D video, and audio based on the relative speed information and mutual position and orientation information between the cards. By synthesizing the generated video signal, video output and audio output can be realized with higher accuracy.

  The preferred embodiments of the present invention have been described in detail above, but the present invention is not limited to such specific embodiments, and various modifications, within the scope of the gist of the present invention described in the claims, It can be changed.

It is a figure of an example which shows the positional relationship of a card | curd and a camera. It is a figure which shows the example of 1 structure of the image | video synthetic | combination output system in this invention. It is a figure which shows an example of the video selection data each accumulate | stored in the moving image output part and the three-dimensional moving image output part. It is a figure which shows one structural example of the audio | voice reproduction | regeneration output system in this invention. It is a figure which shows an example of the data which an audio | voice reproduction | regeneration output part has. It is a figure which shows the example of 1 structure of the image | video synthetic | combination / audio | voice reproduction | regeneration output system in this invention. It is a figure which shows an example of the data for selecting the image | video accumulated in the moving image output part, the three-dimensional moving image output part, and the audio | voice reproduction output part. It is a figure which shows an example of the video output based on the data example of FIG. 7, and an audio | voice output.

Explanation of symbols

11, 21, 41, 61, 81 Card 12, 22, 42, 62 Camera 23, 43, 63 Card information acquisition device 24 Video composition output device 25, 45, 65 Speed detection unit 26, 67 Video image output unit 27, 68 3D moving image output unit 28, 70 Video composition output unit 44 Audio reproduction output device 46, 68 Audio reproduction output unit 64 Video synthesis / audio reproduction output device 66 Mutual position / attitude detection unit

Claims (9)

In a video composition output device for synthesizing and outputting another video signal to the video signal based on the card information consisting of the card type, three-dimensional position information, and posture information from the video signal of the camera that shoots the card ,
A speed detector that detects the speed of the card based on the card information obtained for each frame at a predetermined frame interval of the video signal captured by the camera;
A video composition output device comprising: a video output unit that selects and outputs a video signal to be synthesized based on the card information and speed information obtained by the speed detection unit. The video output unit
Based on the card information and the speed information obtained by the speed detection unit, select a moving image, and based on the three-dimensional position information and the posture information, display, non-display, size of the moving image, The video composition output device according to claim 1, wherein the video composition output device controls and outputs a position and a direction. The video output unit
Based on the card information, speed information obtained by the speed detector, color information and connection information obtained from the three-dimensional position information, a three-dimensional moving image is selected, and the three-dimensional position information and the posture information are selected. The video composition output device according to claim 1, wherein the display, non-display, size, position, and direction of the three-dimensional moving image are controlled based on the output. When there are a plurality of cards in the video signal captured by the camera,
The video composition output device according to any one of claims 1 to 3, wherein the speed detection unit detects relative speed information between the plurality of cards based on the card information.   5. The video composition output device according to claim 4, further comprising a mutual position / posture detection unit configured to detect mutual position / posture information including distance between cards and posture information from card information of the plurality of cards. The video output unit
6. The video composition output device according to claim 5, wherein the video image and / or the three-dimensional moving image is selected and output based on the relative velocity information and the mutual position and orientation information. In an audio reproduction output device that reproduces and outputs pre-stored audio based on card information including card type, three-dimensional position information, and posture information from a video signal of a camera that shoots the card,
A speed detector that detects the speed of the card based on the card information obtained for each frame at a predetermined frame interval of the video signal captured by the camera;
Wherein based on the velocity information obtained by the card information and the speed detection unit, possess an audio reproduction output unit for reproducing and outputting selected audio file,
When there are a plurality of cards in a video signal photographed by the camera, the speed detection unit detects relative speed information between the plurality of cards based on the card information. . 8. The audio reproduction output device according to claim 7 , further comprising a mutual position / posture detection unit configured to detect mutual position / posture information including card distance and posture information from the card information of the plurality of cards. The audio reproduction output unit
9. The audio reproduction output device according to claim 8 , wherein the audio file is selected and output based on the relative speed information and the mutual position and orientation information.

Images