JP4548313B2 - Video creation device and video creation method - Google Patents

Description

  The video creation apparatus and the video creation method for creating a motion image including a motion image of a meaningful person, particularly including a motion image of a person that can be associated with words, sentences, symbols, etc., such as sign language. The present invention relates to a video creation apparatus and a video creation method for creating a continuous motion video by connecting a plurality of motion videos and combining a part thereof.

  When synthesizes a moving person image, it records a video of a series of operations, and synthesizes the image of the person in those image frames into each image frame of the CG image, thereby creating a live-action image of the person in motion Was synthesized on the CG video. However, when a single motion video is connected and combined to generate a continuous motion video, the person's position and shape are shifted, resulting in an unnatural motion video. In order to solve this problem, synthesis was performed on a CG video using a video recording a series of human actions. In this case, the continuous motion video is reused in another scene on the CG video. I could not.

In order to solve this, the invention of Patent Document 1 has been proposed. In order to dynamically create highly realistic video content, the present invention prepares a plurality of live-action video files in which, for example, an actual person's movement is photographed, and an intermediate video portion is prepared from them as necessary. Are synthesized and then linked.
JP 2003-69900 A

  However, when the above video files are simply combined, there is a problem that the distortion of the video caused by the overlapping of each part of the human body such as the face and hand is noticeable and gives an unnatural impression.

  In addition, it is necessary that all persons operating on the video used for composition are the same person, and there is a problem that the burden on the individual increases when shooting the video.

The present invention has been made in view of the above problems, and by holding and using the motion images of each part of a person's body in the images as separate data, it is possible to reduce noise mixed during synthesis, etc. And try to create high quality continuous video. That is, a video containing only the motion of each part of the body, such as a video of the right hand area, video of the left hand area, video of the face area, etc. of the person being photographed is divided into words as individual video files , and motion videos are stored. The metadata registered in the means and divided for each word is registered and used in the connection information storage means. Furthermore, when synthesizing the intermediate images of each motion, the intermediate images of each body region are synthesized, and then a composite image is created by superimposing each video as the final human motion. The aim is to create a high-quality sign language image by reducing the distortion of the composition and the noise mixed during the composition.

  In addition, because data is held by dividing each part of the body such as the hand area and face area, the partial data such as the hand area of a specific person can be used in combination with the partial data such as the face area of another person. It is intended to expand the range of possible images.

To this end, the video creation device of the present invention is a video creation device that creates a continuous motion video by linking a plurality of motion images representing the motion of the body according to specific information, and the motion for each part of the body Motion video storing means for individually storing the motion video each of which is included, information on each of the motion video, read information of the motion video, and one or a plurality of connection criteria displayed in the motion video The link reference information storage means for storing the information in association with each other, and the corresponding read-out information of the motion video is searched from the link reference information storage means based on a plurality of pieces of input information in time order, and the read information is Based on the motion video acquisition means for extracting the motion video from the motion video storage means based on the motion video of the plurality of body parts extracted by the motion video acquisition means, Using a video generating means for generating the operation image of the body by superimposing the operation image of the body part, the image information of a plurality of processing frame obtained by decomposing the operation image of a specific person, recognition, divided, each body Video segmentation means that creates body region frames for each part, accumulates them with time information, and creates a body region file for each body part, and recognizes the characteristics of the motion video of each body part to generate connection reference information Based on a connection reference information means, an information dictionary prepared in advance, information recognizing means for recognizing one or a plurality of information represented by the motion video, and motion video of each recognition information recognized by the information recognition means divided from the division operation picture and the recognition information the operation video storage means linking criterion information corresponding to and, and a registration means for registering to the linking criterion information storage means It said image segmentation means divides each portion corresponding to a word time-series data consisting of the connection reference information and the time information, the body region file per each body part using the time information of the time series data, separated Dividing into words, the registration means registers the body region file divided into words into the motion video storage means, and registers the connection reference information divided into words into the connection reference information storage means. characterized in that it.

Furthermore, the video creation method of the present invention is a video creation method for creating a continuous motion video by connecting a plurality of motion videos representing body movements according to specific information in a video creation device , comprising motion video storage means In the operation video storing step of individually storing the operation images each including the operation of each part of the body, and the connection reference information storage means, the information of each operation image is included in the information of the operation images. And a connection reference information storing step for storing one or a plurality of connection reference information displayed in the motion video in association with each other, and the motion video acquisition means based on a plurality of pieces of input information in time order. Find the read information of the operation image corresponding from the connection reference information storage unit, and operates the image acquisition step of taking out the operation image from the operation image storage means on the basis of the read information, those A plurality of said operation image of each body part extracted in operation the image acquisition process connected with the image forming means, and the image generation step of generating the operation image of the body by superimposing the operation image of each body part linked, image segmentation By using the image information of a plurality of processing frames obtained by disassembling the motion video of a specific person by means, it recognizes and divides it into body region frames for each body part, and accumulates them together with time information for each body A video segmentation step for creating a body region file for each part, and a connection criterion information unit that recognizes the characteristics of the motion image of each body part and generates connection criterion information, and an information recognition unit in advance. Based on the prepared information dictionary, an information recognition step for recognizing one or a plurality of information represented by the motion video, and the motion video of each recognition information recognized in the information recognition step are separated from the body part information. And the registration step of registering the segmented motion image and the connection reference information corresponding to the recognition information for use in the operation image storage step and the connection criterion information storage step by a registration means, and the image division step The video dividing means divides the time series data composed of the connection reference information and the time information into parts corresponding to words, and uses the time information of the divided time series data for the body region file for each body part. In the registration step, the registration means registers the body region file divided for each word in the motion video storage means, and the connection reference information divided for each word is used as the connection reference. The information is stored in the information storage means .

  When video synthesis is performed, the distortion caused by the overlapping of body parts can be eliminated, and the video can be synthesized in a more natural form.

  In addition, since the images of each part of the body, such as the images of the hand area and the face area, can be combined and used, the effect that the range of usable images is widened can be obtained.

  Further, by adding the video dividing unit to the system, the operator does not need to perform the video dividing operation, so that the worker can easily add words and video files to the system.

  Next, embodiments of the present invention will be described with reference to the accompanying drawings.

[First Embodiment]
First, a first embodiment of the present invention will be described. In the present embodiment, explanation will be given by taking sign language as an example. Specifically, an example in which the present invention is applied to a video display device that displays a sign language expressing the meaning of an input sentence as a continuous video will be described with reference to the drawings.

  FIG. 1 is a system configuration diagram of a video display apparatus according to an embodiment. As shown in FIG. 1, the video display device includes an input unit 10, a morphological analysis unit 11, a translation unit 12, a data selection unit 13, a word dictionary 14, a selected video file 15, a display unit 16, a synthesis unit 17, and a video file group. 18 is provided.

  The input unit 10 captures a sentence meaning that the user wants to display an image (in this embodiment, Japanese), and gives the captured sentence to the morphological analysis unit 11. Examples of realizing the function of the input unit 10 include various types such as character input reception by a keyboard, voice text input reception in which voice input from a microphone is converted into text, or data stream input reception by broadcast radio waves captured by an antenna. Input form is possible. That is, all means (input means using operation of buttons, voice, radio waves, etc.) that allow the user to input the contents of the text that the user wants to display video can be used.

  The morphological analysis unit 11 receives an input sentence from the input unit 10, performs morphological analysis on the input sentence, decomposes the input sentence into morphemes, and performs a series of processes for assigning parts of speech to the morphemes using a dictionary (not shown). Part. Further, the morpheme analysis unit 11 gives the analyzed word to the translation unit 12.

  The translation unit 12 receives the word analyzed by the morpheme analysis unit 11 and performs processing such as changing the word order of words, removing unnecessary words, and adding necessary words in accordance with word information and word part-of-speech information. is there. Also, the translation unit 12 gives the processing result to the data selection unit 13.

  The data selection unit 13 is a part that receives a processing result (adjusted word) from the translation unit 12 and searches for a video file necessary to express the meaning of the word. When the data selection unit 13 receives the adjusted word from the translation unit 12, the data selection unit 13 reads out information associated with the word from the word dictionary 14 (for example, a video file name, metadata in the video, which will be described later, And the video file necessary for expressing the meaning of the word is searched from the video file group 18 using the video file name, metadata, and the like as search keys. Further, the data selection unit 13 stores the searched video file in the selected video file 15.

  In addition, when the input word is not registered in the word dictionary 14, the data selection unit 13 searches the word dictionary 14 for an image showing a finger character expressing the sound of the word in sign language, and gives the selected image file 15. .

  The data selection unit 13 retrieves the corresponding motion video read information from the word dictionary 14 based on a plurality of input information in time order, and the motion video is retrieved from the video file group 18 based on the read information. An operation video acquisition means for extracting is configured.

  Here, the video file will be described with reference to FIG.

  A video file is a video file that is divided into parts that express the meaning of words. In other words, a video file is a sign language in which only the face is reflected, only the right hand is reflected, and only the left hand is reflected, for example, by processing the part other than the reflected part to be the same color as the background at the time of shooting. A person is photographed, and the photographed video is separated by a portion expressing the meaning of each word, and a name is given to each of the separated portions. These video files are called a face video file, a left hand video file, and a right hand video file, respectively.

  The video file is configured by continuously arranging a plurality of still images, and each still image is called a frame. In FIG. 2, 21, 22 and 23 show examples of frames of a face video file, a left hand video file and a right hand video file, respectively. Reference numeral 24 denotes an example in which the frames of the face video file, the right hand video file, and the left hand video file are displayed as a video of a person having a complete frame.

  The face video file group 18F, the right hand video file group 18R, and the left hand video file group 18L in FIG. 1 are an aggregate of a plurality of video files in which only the areas of the respective names are shown. Furthermore, the face video file group 18F, the right hand video file group 18R, and the left hand video file group 18L are aggregates of a plurality of video files that are finally displayed on the display unit 16 and are elements constituting a continuous video, It is an aggregate of a plurality of video files including videos in which motions corresponding to finger character motions, which are motions for expressing each Japanese word to be expressed and Japanese sounds in sign language, are captured. These are collectively referred to as a video file group 18. The video file group 18 constitutes motion video storage means for individually storing motion videos each including the motion of each part of the body.

The word dictionary 14 stores each word in association with a video file name of a video file including a video representing the meaning of the word, an operation duration in the video file, metadata in the video, and the like. Storage area. Note that the word dictionary 14 can be newly registered, added, and deleted. In this word dictionary 14, connection criterion information storage means for storing the information of each motion video in association with the readout information of the motion video and one or a plurality of link criterion information displayed in the motion video. Is configured.

  Table 1 shows a configuration example of the word dictionary 14 of the present embodiment. As shown in Table 1, the word name indicates the name of a registered word. Reading indicates the reading of the word. The part of speech indicates the part of speech name of the word. The file name indicates the name of the video file representing the word. There are three video file names for face, left hand, and right hand. The creation date represents the date when the video file was created and the data was registered in the word dictionary. The duration indicates the duration of the operation in the video file, and the unit is second.

  In Table 1, metadata is data extracted from video file contents necessary for video composition and connection. The configuration of the metadata can be different depending on the content of the video file and the content of the video to be output. In this embodiment, since the output video is used as sign language, the word dictionary 14 stores the direction of the person's line of sight, mouth shape, and human hand in each frame of the three types of sign language video representing the meaning of the input word. The position, hand shape, hand orientation, velocity vector, etc. are stored as metadata.

  In Table 1, the start point indicates the metadata of the first frame of the video file, and the end point indicates the metadata of the last frame of the video file. The metadata of other frames has the same structure as the metadata of the first and last frames.

  Next, each item in the metadata of Table 1 will be described with reference to FIG. The position of the hand in Table 1 represents the position of the center of gravity of the hand area of the person appearing in the video, expressed by two-dimensional coordinates in the video. Although not shown in Table 1, an operation speed vector obtained by differentiating the hand position with respect to time may be used.

  FIG. 3 shows an example of a screen that is actually displayed. The vertical direction of the screen is the y-axis and the horizontal direction is the x-axis. The intersection of the y axis and the x axis is defined as a reference point (0, 0). In this case, the position of the right hand in the frame can be the coordinates (x, y), and the position of the left hand can be (x0, y0).

  The shape of the hand in Table 1 indicates the shape of a finger such as a person extending or bending a finger of the hand. Usually, about 80 types of hand shapes are distinguished in sign language. In the present embodiment, symbols representing the distinguished hand shapes are determined and used. FIG. 16 shows some examples of symbols corresponding to hand shapes.

  The direction of the hand in Table 1 is a combination of the direction in which the hand is facing and the direction in which the palm is facing. Here, the direction in which the hand is facing is the direction in which a straight line drawn from the elbow of the person to the wrist is facing. In normal sign language, there are about 20 to 28 types of directions facing the hand and about 6 to 8 types of palm orientations. Therefore, the direction types are represented by symbols, and those symbols are shown in the table. The combination of is written.

  FIG. 4, FIG. 5 and FIG. 6 show the correspondence between the direction of the hand and the symbol, and FIG. 7 shows a part of the example of the correspondence between the palm direction and the symbol.

  FIG. 4 shows a direction on a plane viewed from the front, that is, a so-called frontal plane. For example, the direction in which the person's right hand is pointing in FIG. The x-axis and y-axis in FIG. 4 are the same as those described in FIG.

  FIG. 5 shows six directions on a plane as seen from the side, that is, a so-called sagittal plane. The direction in which the person's right hand is pointing in FIG. The z-axis in FIG. 5 is an axis perpendicular to the x-axis and the y-axis in FIG. 4, and is an axis that is directed from the person's body toward the face.

  FIG. 6 shows four directions on a plane when a person is viewed from directly above, a so-called horizontal plane. The direction in which the person's right hand is pointing in FIG. 6 is represented as a16. Each axis in FIG. 6 is the same as that in FIGS.

  The axis v in FIG. 7 is an axis indicating the direction in which the hand is facing as described above. The thin arrows are arrows arranged on a plane perpendicular to the axis v and indicate the direction in which the palm is facing. The direction in which the palm faces in FIG. 7 is represented as aG.

  The direction of the line of sight in Table 1 indicates the direction in which a person is looking. FIG. 14 shows the correspondence between the line-of-sight direction and the coordinate system. The method of expressing the direction is the same as that shown in FIGS. Each axis in FIG. 14 is the same as that in FIGS. The direction of the line of sight of the person in the figure is represented as a10.

  The mouth shape in Table 1 indicates the shape of a person's mouth. In sign language, vowels are often used as clues, so vowels that represent the mouth are used as symbols. FIG. 15 shows an example of correspondence between mouth shapes and symbols.

  In this embodiment, the metadata is configured as described above in order to display the sign language image. However, it is one of the information of the image itself such as the information of the contents expressed by the image, the information of the image frequency, and the color histogram. The part may be used as metadata.

  The face selection video file 15F, the left-hand selection video file 15L, and the right-hand selection video file 15R in FIG. 1 are for each of the areas shown in FIG. 2 for each word searched by the data selection unit 13. This is a storage area for receiving the video files and holding these video files for each area. These are collectively referred to as a selected video file 15.

  The synthesizing unit 17 extracts two adjacent video files for the face selection video file 15F, the left-hand selection video file 15L, and the right-hand selection video file 15R in the stored selection video file 15, respectively. The intermediate video file is synthesized. In the combining unit 17, a plurality of motion images of the respective body parts taken out by the data selection unit 13 are connected, and a video creation means for creating a motion image of the body by superimposing the motion images of the connected body parts; Based on the plurality of motion images that are acquired by the data selection unit 13 and that move forward and backward, an intermediate motion image is created for each motion image of each body part, and is combined with the video creation means.

  The display unit 16 is a functional unit that performs processing of displaying a plurality of selected video files 15 (F, L, R) as appropriate and continuously displaying them.

[Video creation method]
Next, a video creation method using the video display device having the above configuration will be described with reference to the drawings. FIG. 8 is a flowchart of a video creation operation by the video display device of this embodiment.

[1. input]
In FIG. 8, first, the user inputs a sentence expressed in sign language video to the input unit 10 (s80, s81, s82). Among these, in s80, the user inputs a sentence having a meaning to be displayed as a sign language image by a sentence input unit such as a keyboard. In some cases, a sentence is input by voice input or the like. In this case, after inputting a sentence by voice input or the like (s81), by converting the voice text (s82), the input voice is converted into a sentence and used as an input to the next step.

[2. Assign to Word]
The morpheme analysis unit 11 performs morphological analysis on the input sentence and decomposes it into morphemes (s83). Next, parts of speech are assigned to each decomposed morpheme using a general Japanese dictionary (not shown) (s84). Here, morphemes to which parts of speech are assigned are referred to as words below. This word is given to the translation unit 12.

[3. Parsing and reordering]
The translation unit 12 parses the sentence divided into words (s85), and changes the order according to the grammar expressed as the state transition between words as in the existing research (s86).

[4. Create selected video file]
[4-1 Search for files]
The data selection unit 13 reads one word, searches the word dictionary 14 using the word name and the assigned part of speech as a key, and checks whether there is a registered word (s87).

  If a word is not registered, a finger character file search process is performed to search for a video file that captures a finger character that expresses the sound of the word in sign language (s88).

[4-1-1 Finger Character Search Processing]
Details of the finger character file search process (s88) will be described with reference to FIG.

  First, the sound of the input word expressed by finger characters is decomposed into character units (sb881), and the characters constituting the word are read one by one (sb882). Next, the name of the file representing the finger motion corresponding to each character is retrieved from the word dictionary 14 and read (sb 883), and the file name is stored in finger character data (sb 884). Next, it is determined whether or not file name accumulation has been completed for all characters (sb885). Thus, the processing is repeated until the processing of sb882 to sb884 is completed for all characters, and finally the file name to be called is determined using the finger character data. (Sb886).

  Next, the file is read from the video file group 18 (sb887), and the read video file is added to the selected video file 15 (s813).

  Returning to FIG. 8, if it is determined in s87 that there is a registered word, it is further determined whether or not there are a plurality of input words (s89). When the number of corresponding cases is one, the process proceeds to s812, and the file of each area is read from the video file group 18 using the file name item of the search result (s812), and the selected video files 15F, 15R, 15L are read. Is added for each region (s813).

[Selection when there are multiple 4-2 registrations]
On the other hand, if it is determined in s89 that there are a plurality of corresponding cases, the metadata in the word dictionary is referred to (s810), and a file to be read from the plurality of candidates is determined (s811). Specifically, the data selection unit 13 refers to the metadata of the file representing the previous word in s810. As a specific example when the metadata can be referred to, the processing will be described using the data in Table 1 as an example.

  The previous word is “Tomorrow” and “01F.avi”, “01L.avi”, and “01R.avi” are selected as video files representing the operation of the word, respectively, and the current word is “Meet” Think about the case.

  As shown in Table 1, since there are a plurality of right-hand video file candidates representing “Meet”, it is necessary to select which file to use. First, the coordinates of the right hand position and the left hand position at the end of the metadata of the files “01F.avi”, “01L.avi”, “01R.avi” representing the previous word “Tomorrow” are obtained ( sb810).

  Then, the coordinates of the right hand position and the left hand position at the time of starting are obtained from the metadata of a plurality of candidate files representing “meet”.

  Here, the left and right sums of the Euclidean distances of the hand position at the end and the hand position at each start are obtained, respectively, and the file with the smallest value is determined as the selected file (s811).

In the case of “04F.avi”, “04R.avi”, and “04L.avi” in this example, the sum of the Euclidean distances of the left and right hands is ((50-50) ∧2 + (50-60) with the right hand. ∧2) ∧ (1/2) = 10
With the left hand ((100-80) ∧2 + (90-60) ∧2) ∧ (1/2) = 36.06. .
Here, the symbol ∧ represents a power.

The left and right sum is 46.06
It becomes.

In the case of “02F.avi”, “02R.avi”, “02L.avi”, ((50-100) ∧2 + (50-60) ∧) ∧ (1/2) = 50.99 with the right hand.
((100-130) １００2+ (90-60) ∧2) ∧ (1/2) = 58.31. .
The left and right sum is 109.30
So that
“04F.avi”, “04L.avi”, “04R.avi”
Is selected.

  If the file cannot refer to the metadata, the data with the new update date is used as the output result of s811.

  In this embodiment, the information on the right hand and the left hand is used as a reference. However, depending on the configuration, it is also possible to use the similarity in the direction of the line of sight and the shape of the mouth.

  Thereafter, the data selection unit 13 reads the selected video file from the video file group 18 (s812), and adds the read video files to the selected video file 15 (F, R, L). (S813).

  The above steps s87 to s813 are repeated for all words constituting the input sentence.

[5. Intermediate file composition]
(Change selected video file)
FIG. 10 shows the relationship between the processing flow of the synthesis unit 17 and the operation flow.

(A. Extraction)
The synthesizing unit 17 represents, for example, a destination video file 97 representing a preceding word that has not yet been processed from the selection video file 15R in the right-hand region, and a word immediately thereafter, out of each region video file of the selection video file 15. Two of the subsequent video files 99 are extracted (s91).

  Next, the last frame 98 of the previous video file 97 and the first frame 910 of the subsequent video file 99 are extracted (s92).

(B. Intermediate frame synthesis)
Next, a plurality of intermediate frames 911 are synthesized by morphing processing using the final frame 98 and the leading frame 910 (s93).

  Here, as a morphing process, it is possible to use an existing technique such as cross dissolve or warping. A frame that gradually changes from the last frame 98 to the first frame 910 is synthesized.

(C. Create video file)
Next, a composite video file 912 is created by concatenating a plurality of intermediate frames synthesized in s93. In other words, time information is added to a plurality of intermediate frames and collected as a video file 912 (s94).

(D. Addition)
Next, a process of inserting the composite video file 912 into the relevant part of the selected video file, that is, an intermediate part between the previous video file 97 and the subsequent video file 99 is performed (s95).

  Next, the above-described a. b. c. d. Repeat each process. After the right hand processing is completed, the same processing is performed for the left hand and face selection video files 15L and 15F. The order of the body parts may be different.

[6. display]
The display unit 16 draws the selected video file so as to overlap with 15F, 15R, and 15L, and finally displays the selected video file as a video of a person with a face, right hand, and left hand as shown in FIG.

A display example of the video created in this way is shown in FIG. FIG. 13 is an example of a display device that displays a sentence and simultaneously translates the meaning of the sentence into sign language and displays it.

  This display device includes a display unit 1201 for displaying information such as train operation status and the video display device 1202 having the above-described configuration. The display unit 1201 automatically displays the current train operation status and the like. The video display device 1202 translates the content of the display unit 1201 into sign language by the above processing, and displays it as a sign language video simultaneously with the text on the display unit 1201.

[effect]
By storing the motion image of each body part as an independent video file, performing the video composition process independently, and displaying it at the end, it is possible to display the body when synthesizing the video in the normal configuration. Distortion caused by overlapping of parts can be eliminated, and an effect of enabling more natural video composition is obtained.

  In addition, since the image of the hand area and the image of the face area can be used even if they are from different people, the range of usable images can be expanded.

[Second Embodiment]
Next, a second embodiment of the present invention will be described. In this embodiment, a video dividing unit 101 is added to the system of the video display device. By adding a video segmentation unit 101 to this system, an image file that captures a person expressing a sign language sentence is analyzed, segmented, and only the body regions such as the face, right hand, and left hand are displayed. A system will be described in which a video file corresponding to a word is divided and stored in a video file group after being divided as a video file, and metadata necessary for searching a word dictionary can be added.

[Constitution]
The configuration of the second embodiment is different from the first embodiment in that a video dividing unit 101 is newly added. Therefore, in FIG. 11, the same or corresponding components as those in FIG. 1 are denoted by the same reference numerals. In addition, description of the functions of the configuration requirements described in the first embodiment is omitted.

  The video segmentation unit 101 analyzes a video file that captures a person representing a sign language sentence, uses image processing techniques such as segmentation, image recognition, and area segmentation, and the person's face, right hand, left hand, etc. A function of dividing the video files into independent video files in which only the body regions are displayed, storing the video files in the video file group 18, and adding metadata necessary for the search to the word dictionary 14 is provided. Functional part. The video dividing unit 101 has a processing function shown in FIG.

  In this video dividing unit 101, using the image information of a plurality of processing frames obtained by decomposing a motion image of a specific person, it is recognized and divided into video frames for each body part, which are accumulated and stored in each body Based on a video segmentation means for making a motion image for each part, a connection reference information means for generating connection reference information for motion images of each body part, and an information dictionary prepared in advance, 1 or Information recognition means for recognizing a plurality of information, and a motion video of each recognition information recognized by the information recognition means is divided from the body part information, and the divided motion video and the connection reference information corresponding to the recognition information are The moving image storage means and the registration means for registering in the connection reference information storage means are configured. Specifically, this is realized by the following processing functions.

[Video creation method]
Next, a video creation method using the video display device having the above configuration will be described. Since the video creation method by the video display device of this embodiment is the same as that of the first embodiment except for the video dividing unit 101, only the operation of the video dividing unit 101 will be described here.

  In FIG. 12, a video file 1101 is a video file obtained by photographing a person expressing a certain sentence in sign language. Hereinafter, a case will be described in which the video file 1101 is divided into body regions of the face, right hand, and left hand, and further divided into words, and records are added to the video file group 18 and the word dictionary.

  First, the video dividing unit 101 takes in a video file 1101 to be added, and divides a plurality of video frames constituting the video file 1101 into a predetermined frame group 1103 (s1102). The frame group 1103 is a group of a plurality of still images.

  When the video file 1101 is divided into the frame group 1103, the video dividing unit 101 extracts one frame (s1104) and sets it as a processing frame 1105.

  The extracted processing frame 1105 is divided into body regions (s1106).

  Only one body part appears from each processing frame using existing methods, such as face region extraction by pattern matching using wavelet transform, arm region extraction by combination of color space conversion and optical flow extraction, etc. Area frames 1107, 1108, and 1109 are created.

  Next, for the face area frame 1107, the right hand area frame 1108, and the left hand area frame 1109 where only each body area exists, the center-of-gravity position detection, which is an existing algorithm, is performed (s1110), and the position of each area is determined. It is added as position data 1111.

  Further, a plurality of feature points are detected in the face region for the face region frame 1107, a graph is formed in the feature point group, and a process of recognizing the direction of the line of sight from the classification of the feature graph is performed (s1112).

  Further, mouth shape recognition and classification are performed by pattern matching or the like in the mouth peripheral region (s1114), and each is added as time series data 1117 (s1116).

  Similarly, image recognition processing and data addition are performed for the right-hand and left-hand region frames (1108, 1109).

  Using the information of the position data 1111, the direction of the hand is recognized from the positional relationship between the hand and elbow of the person in the processing frames 1108 and 1109, and the recognized hand direction is classified into symbols (s 1113). Next, pixel information of a certain area around the recognized hand position is acquired, the shape of the hand is recognized using the pixel information, and the recognized hand shape is classified into symbols (s1115). At this time, as a method for recognizing the shape of the hand, for example, for a pixel with a certain area around the position of the hand, a method such as neural network or calculation of higher-order local autocorrelation features, which is an existing image recognition algorithm Can be applied.

  In this way, the video dividing unit 101 adds information indicating the display time of the processing frame 1105 in the face area frame 1107, the right hand area frame 1108, and the left hand area frame 1109, and thereby the direction of the line of sight, mouth shape, and hand. Is added to the time-series data 1117 (s1116).

  Further, the face area frame 1107, the right hand area frame 1108, and the left hand area frame 1109 are added to the face area file 1123, the right hand area file 1124, and the left hand area file 1125 (s1118).

Here, the face area file 1123, the right-hand area file 1124, and the left-hand area file 1125 are storage areas that have frame and time information, respectively, and accumulate as general video files.

  Table 2 shows an example of the structure of time series data. As shown in Table 2, time-series data includes metadata (gaze direction, mouth shape, hand position, hand direction, hand shape) for each predetermined time (here, the time unit is milliseconds). ).

  Then, the processing from S1104 to S1118 is repeated until all the frames in the frame group 1103 are completed.

  At this time, various kinds of metadata are stored in the time series data 1117, the face area file 1123 is an image of only the face, the right hand area file 1124 is an image of only the right hand, and the left hand area file 1125 is an image of only the left hand. Stored as a file.

  When the processing is completed for all frames, the video segmentation unit 101 recognizes an existing recognition such as an HMM that inputs the position of the hand, an operation speed vector that is a time derivative thereof, a hand peeling, etc. with respect to the generated time series data. Using the method, sign language recognition processing is performed, and time-series data is divided into portions corresponding to sign language words (s1119).

  Further, the video dividing unit 101 attaches a label of the sign language word name of the recognition result to the divided time series data (s1120), reads the face area file, the right hand area file, and the left hand area file and labels the time series data. Each time, using the time information of the time-series data, the face area file, the right hand area file, and the left hand area file are divided for each label (s1121).

  Then, the video dividing unit 101 names the divided face area file, right hand area file, and left hand area file, and the face video file group 18F, the right hand video file group 18R, and the left hand video file group 18L of the video file group 18, respectively. Is added as a record (s1122), and metadata is added to the word dictionary 14.

[effect]
In the present embodiment, the following effects are obtained in addition to the effects of the first embodiment.

  Up to now, it has been necessary to devise at the time of shooting, such as shooting a person's actions a plurality of times by masking each area at the time of system creation, but this embodiment can solve this. That is, by adding the video dividing unit 101 to the system, it is not necessary for the worker to perform the video dividing operation, and as a result, the worker can easily add words and video files to the system. .

[Modification]
In each of the above embodiments, a method such as a keyboard is used as the input unit. However, any input device such as a voice input, a GUI-like interface, a button, or a barcode reader can be used.

  In each of the above embodiments, the video file group 18 and the word dictionary 14 have been described as being configured as separate parts. However, the present invention is not limited to this, and other configurations can be used as long as they have the same function. But you can. For example, metadata may be embedded in a video file header or the like, and a header of a specific video file group may be searched as needed as a word dictionary.

  In each of the above embodiments, in order to express sign language in the video file metadata, the direction of the line of sight, the shape of the mouth, the position of the hand, the direction of the hand, the shape of the hand, but these are examples, The present invention is not limited to these. For example, another data such as facial expressions may be used.

  In the first embodiment, the Euclidean distance is used in the evaluation criteria for selecting a video file. However, this is an example, and another value that can obtain the same effect may be used.

  In the first embodiment, as examples of morphing methods, cross dissolve and warping are given as examples. However, these are not exclusive, and either one or both may be used in combination. Other synthesis methods may also be used.

  In each embodiment, a video display device that displays a sign language video has been described, but not limited to sign language, a plurality of videos including actions of a person having a meaning that can be associated with words, sentences, and symbols, Any device that displays continuous human action images can be used as a system for displaying action images for expressing a specific meaning, such as a hand flag signal, pantomime, and dance.

  In each embodiment, a dictionary search in Japanese is performed, but it is also possible to make a sign language display system in another foreign language by changing the dictionary data.

  In each embodiment, the configuration of each dictionary is shown as a table, but a different data structure such as a tree may be used.

  The elements such as storage means constituting the present invention may be any hardware configuration. For example, it may exist on a single PC, or a part or all of elements such as a database may be a server on a network.

  Further, in each of the above embodiments, the video display device including the display unit 16 has been described as an example, but a video creation device that does not include the display unit 16 can exhibit the same operations and effects as described above.

1 is a system configuration diagram of a video display apparatus according to a first embodiment of the present invention. It is a schematic diagram which shows the example of the video file of each body part. It is a schematic diagram which shows the position of the hand among each body part. It is a schematic diagram which shows direction of the hand on a frontal surface. It is a schematic diagram which shows the direction of the hand on a sagittal surface. It is a schematic diagram which shows direction of the hand on a horizontal surface. It is a schematic diagram showing the direction of the palm. It is a flowchart which shows the creation process of the selection video file. It is a flowchart which shows the creation process of a finger character video file. It is a block diagram which shows the intermediate data produced | generated by the flowchart which shows the process in a synthetic | combination part, and a synthetic | combination part. It is a system configuration | structure figure of the video display apparatus concerning the 2nd Embodiment of this invention. It is a flowchart which shows the process in a video division part. It is a front view which shows the usage example of an imaging | video production apparatus. It is a schematic diagram which shows the correspondence of the direction of eyes | visual_axis and a symbol. It is a schematic diagram which shows the correspondence of the shape of a mouth and a symbol. It is a schematic diagram which shows the correspondence of a hand shape and a symbol.

Explanation of symbols

  10: input unit, 11: morphological analysis unit, 12: translation unit, 13: data selection unit, 14: word dictionary, 15: selection video file, 15F: selection video file of face, 15L: selection video file of left hand, 15R : Right hand selected video file, 16: display unit, 17: compositing unit, 18: video file group, 18F: face video file group, 18R: right hand video file group, 18L: left hand video file group, 101: video dividing unit.

Claims (9)

A video creation device that creates a continuous motion video by linking a plurality of motion images representing physical motions according to specific information,
Motion video storage means for individually storing motion videos each including motion for each part of the body;
Linked reference information storage means for storing the information of each moving image in association with the read information of the moving image and one or more pieces of linked reference information displayed in the moving image;
A motion video that retrieves the corresponding motion video readout information from the connection reference information storage means based on a plurality of temporally input information and extracts the motion video from the motion video storage means based on the readout information Acquisition means;
A video creation means for connecting the motion images of each of the body parts taken out by the motion video acquisition means, and superimposing the motion images of the connected body parts to create a motion video of the body,
Using the image information of multiple processing frames obtained by decomposing a motion image of a specific person, it is recognized and divided into body region frames for each body part, and these are stored together with time information for each body part. Video segmentation means as body region files;
A connection reference information means for recognizing the feature of the motion image of each body part and generating connection reference information;
Information recognition means for recognizing one or more pieces of information represented by the motion video based on an information dictionary prepared in advance;
The motion video of each recognition information recognized by the information recognition means is divided from the body part information, and the motion video storage means and the connection reference information storage means corresponding to the divided motion video and the connection reference information corresponding to the recognition information. And registration means for registering with
The video dividing means divides the time series data composed of the connection reference information and the time information into parts corresponding to words, and uses the time information of the divided time series data to obtain the body region file for each body part. Divide each word above,
The video in which the registration means registers the body region file divided for each word in the motion video storage means, and registers the connection standard information divided for each word in the connection standard information storage means Creation device.   The video creation apparatus according to claim 1, wherein the connection reference information is metadata necessary for composition and connection of each motion video.   Based on the plurality of motion images that are acquired by the motion image acquisition means, the intermediate motion image is generated for each motion image of each body part, and is provided with synthesis means that is delivered to the image creation means. The video creation device according to claim 1 or 2. The above motion image is a sign language image expressing the meaning of a word,
The video creation apparatus according to any one of claims 1 to 3 , wherein the connection reference information is metadata including a hand position of a person who performs sign language, a hand peeling, and a hand shape. 5. The video creation device according to claim 1 , wherein the video creation means includes a display unit that displays the created video. In a video creation device, a video creation method for creating a continuous motion video by connecting a plurality of motion videos representing body motion according to specific information,
An operation image storage step of individually storing operation images each including an operation for each part of the body in the operation image storage means ;
The connection reference information storage means stores the connection reference information in which the information of each operation video is stored in association with the read information of the operation video and one or more connection reference information displayed in the operation video. Process,
The motion video acquisition means retrieves the corresponding motion video read information from the connection reference information storage means based on a plurality of input information in time order, and the motion video storage means reads the motion video storage information based on the read information. An operation image acquisition process for extracting an operation image;
A video creation step of connecting a plurality of motion images of each of the body parts extracted in the motion video acquisition step with a video creation means, and superimposing the motion images of the connected body parts to create a motion video of the body,
Using image information of a plurality of processing frames obtained by disassembling a specific person's motion video by video segmentation means, recognize and divide it into body region frames for each body part, and store them together with time information The video segmentation process to be a body area file for each body part,
A connection reference information step of generating connection reference information by recognizing the characteristics of the motion image of each of the body parts by a connection reference information means,
An information recognizing step for recognizing one or more pieces of information represented by the motion video based on an information dictionary prepared in advance by an information recognizing means;
The motion image of each recognition information recognized in the information recognition step is divided from the body part information, and the motion image storage step and the connection criterion are stored in the registration means by using the divided motion image and the connection reference information corresponding to the recognition information. A registration process for registering for use in the information storage process,
In the video dividing step, the video dividing means divides the time series data composed of the connection reference information and the time information into parts corresponding to words, and uses the time information of the divided time series data for each body part. Split the body region file for each word above
In the registration step, the registration means registers the body region file divided for each word in the motion video storage means, and registers the connection reference information divided for each word in the connection reference information storage means. A featured video creation method. The video creation method according to claim 6 , wherein the connection reference information is metadata necessary for composition and connection of each motion video. Based on the plurality of motion images that are acquired in the motion image acquisition step, the body motion parts are inserted in the middle when the motion images of the body parts are connected in the image creation step. The video creation method according to claim 6, further comprising a synthesis step of creating each motion video. The above motion image is a sign language image expressing the meaning of a word,
9. The video creation method according to claim 6 , wherein the connection reference information is metadata including a hand position of a sign language person, a hand peeling, and a hand shape.

Images