JP2910256B2 - Moving image transmission apparatus and moving image transmission method - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a moving picture transmission apparatus and a moving picture transmission method, and more particularly to an encoding / decoding apparatus for transmitting a moving picture signal with high efficiency.

[0002]

2. Description of the Related Art FIG.
ol. 42, No. 6 (1988), which is a conceptual diagram of a conventional high-performance video encoding / analysis / synthesis coding system, in which 50 is a transmitting unit, 51 is a receiving unit, and 52 is a face. Reference numeral 500 denotes an input image; 510, analysis result information (structure information, motion information, facial expression information, update information); and 520, vertex coordinate information and connection information of a three-dimensional face model. And 530 is composite image output information.

A conventional moving picture encoding / decoding apparatus using the analysis / synthesis encoding method is configured as described above. For example, when a subject is a face image on a shoulder of a person, a three-dimensional model 52 of the face is used. Then, a wire frame model (FIG. 6) stored in advance is matched with the input face image, and processing such as structural analysis and expression analysis is performed based on the movement of a feature point on the wire frame model. As a result, analysis result information 51
0 is transmitted to the receiving unit 51, and the three-dimensional model 5
2 to perform image synthesis processing such as expression synthesis, thereby obtaining synthesized image output information 530.

[0004]

In a moving picture transmission apparatus using the conventional analysis / synthesis coding method as described above, when a subject image, for example, a face image of a person frequently moves, the motion information is analyzed. The result information 510 must be sequentially transmitted, and there is a problem that a significant information amount compression effect cannot be expected.

The present invention has been made in order to solve such a problem, and always reproduces a motion faithful to an original image with a small amount of information even when a complicated motion continuously occurs in a subject image. It is an object to obtain a moving image transmission device and a moving image transmission method.

[0006]

According to a first aspect of the present invention, there is provided a moving image transmitting apparatus having an image input unit for converting an image input from an image input unit into a digitized image signal. Further, it has a frame memory for storing the input image of one frame before. The image processing apparatus further includes a motion detection unit that detects a motion of a subject from the current input image and the input image one frame before using the three-dimensional model. Further, it has a function equation determination unit that approximates the motion of the subject with a function equation, and a function equation database in which a finite number of function equations are registered. The transmission apparatus includes an error value threshold processing unit that calculates a distortion between the detected motion information of the subject and an approximate value of the motion based on the function equation. In addition to the functional equation of the video transmission equipment, its functional equation
The coefficient group of the equation is also registered as a database, and the function equation
The constant section and error value threshold processing section
It is made to be able to handle.

[0007] According to a second aspect of the present invention, there is provided a moving image transmitting apparatus.
2. The moving picture transmission apparatus according to claim 1, wherein
The fixed part is registered in the above function equation / coefficient / database.
Approximate the motion using the function equation and coefficient group
The value threshold processing unit is a function information that gives a more approximated motion.
And send the above function equation / coefficient / database
Video transmission device on the receiving side and video transmission device on the receiving side
By sharing the same functional equation and coefficient group with
Function numbers are transmitted as information indicating function equations and coefficient groups
Video transmission device on the receiving side from the video transmission device on the receiving side
Output to the device to transmit the moving image with less information.
It is characterized in that it can be sent.

[0008] According to a third aspect of the present invention, there is provided a moving image transmitting apparatus comprising:
This is related to the receiving side.
A motion model creation unit that reproduces the motion of the subject image from the dimensional model and the function information transmitted from the transmission side; and a synthesized image output unit that finally outputs a synthesized image of the subject with motion. .

According to a fourth aspect of the present invention, there is provided a moving image transmission method comprising:
The subject corresponding to the time axis in each of the three-dimensional directions
A registration step of registering a plurality of function equations and a plurality of coefficient groups representing the movement of the image as function information as a database, an image input step, and a detection step of detecting movement from the input image. A determination step of selecting and determining function information for approximating the movement from the database; and an error value threshold processing step of checking whether or not the approximation is within an allowable range and restarting the determination step when the approximation is outside the allowable range. .

[0010] According to a fifth aspect of the present invention, there is provided a moving image transmission method comprising:
The present invention relates to a receiving method and includes a model creating step of regenerating a motion based on an output from a transmitting side.

[0011]

In the moving picture transmitting apparatus and the moving picture transmitting method according to the first, second, third, fourth and fifth aspects, the motion of the subject image is detected using the image signals of two consecutive frames. Therefore, extremely stable motion detection can be performed. In addition, since the motion detected as described above can be continuously approximated over a plurality of frames by using a function equation having the smallest error among the function equations registered in a finite number in the database, the amount of transmitted information is always small. Yes. Further, an error threshold processing unit (or an error threshold processing step)
Determines that the motion of the subject has changed greatly and cannot be approximated by the previous function equation. In such a case, the function equation determination unit (or the determination step) newly adds a new function equation Choose the one with the smallest error from
By using this, it is possible to flexibly cope with a severe and complicated movement of the subject.

[0012]

[Embodiment 1] FIG. 1 is a block diagram on the transmitting side showing an embodiment of the present invention. 1 is an image input unit, 2 is a three-dimensional model matching unit, 3 is a frame memory, 4 is a motion detection unit, 5 is a function equation determination unit, 6 is a function equation / coefficient / database, 7 is an error threshold processing unit, 8 Is a three-dimensional model. 90 is a digitized image signal, 100 is three-dimensional model information,
Reference numeral 110 denotes a subject image signal obtained by matching the three-dimensional model, 1
20 is a subject image signal one frame before, 130 is motion information of the subject image after motion detection, 140 is information of a functional equation and its coefficient group, 150 is motion parameter information approximated by the functional equation, and 160 is another new A flag indicating whether or not approximation is performed using a function equation, and 170 is a motion parameter including a function equation number and a coefficient group.

FIG. 2 is a block diagram on the receiving side showing an embodiment of the present invention. Reference numeral 9 denotes a motion parameter 170, a three-dimensional model 8, and a function equation / coefficient database 6 output from the transmitting side. A motion model creation unit that creates a motion of a subject and outputs motion information 190. A synthesized image output unit 10 that finally outputs a synthesized image signal 200 from the motion information 190 and the digitized input image signal 90. It is.

A detailed description of the moving picture communication device configured as described above will be described below with reference to FIGS. First, an input image from a camera or the like is converted into a digital signal in the image input unit 1 and a digital input image signal 90 is input.
Get. Next, a wire frame model stored in the three-dimensional model 8 for the subject based on the digital input image signal 90 (FIG. 5 is a wire frame model above the shoulder of the person, and FIG. 6 is a wire frame model of the face region). And the subject image signal 110 after the matching is obtained. The subject image signal 110 is input to the frame memory 3, and motion detection is performed between the subject image signal 110 and the next frame.

FIG. 3 illustrates this, in which a subject (corresponding to a human head in the figure) 11 in the previous frame has moved to a subject image 12 in the current frame. The function of the motion detector 4 is to detect the motion information such as the rotation angles about the X, Y, and Z axes and the distance of the parallel movement. At this time, it is known that the motion of the subject can be accurately obtained by detecting the motion information of the three feature points in the subject. In the present invention, the detection is relatively easy as shown in FIG. The center of both eyes of the person's head and the top of the nose were determined as feature points.

Next, motion information of the detected subject is represented using an optimal function equation. First, as a concept of the present invention,
If the detected motion information 130 of the subject is decomposed into rotational components around the X, Y, and Z axes, the motion around each axis can be represented on a two-dimensional plane with respect to the time axis. That is, the X axis,
If the movements of the Y axis and the Z axis are represented by Mx, My, Mz, and the time is represented by t, Mx = fx (t), My = fy (t), M
That is, it can be described by a functional equation of z = fz (t).

Hereinafter, only the X-axis will be described for the sake of simplicity. Using the fact that the motion can be represented by a function equation, the function equation determination unit 5 searches the (k + 1) functions in FIG. 7 for the function (optimum function) that best approximates the rate of change of the motion information 130 with respect to the time axis. These k + 1
The function equations and coefficient groups representing them are stored in advance in a function equation / coefficient database. However, if the absolute value L, L = | Mx−Mmin |, of the error between the output value Mmin of the optimal function and the actually detected motion Mx is larger than the threshold value T (L ≧ T), the optimal function Since there is a limit in approximation, another function equation registered in the function equation / coefficient group / database 6 is called to find an optimal function. At this time, the flag 160 for the error value threshold processing unit 7 to search for another function equation is turned ON. Then, the number and coefficient group of the searched optimal function (FIG. 7)
, The set A _k ) of P _k pieces is the motion parameter 1
70 is transmitted to the receiving side.

On the other hand, if L <T in the original frame following the previous frame, the optimal function and its coefficient group used in the previous frame can be used as they are, and the flag 16
0 remains OFF. At this time, no motion parameter 170 is output.

FIG. 8 is a diagram showing the flow of the above processing in relation to the movement and time. A dotted line 130 corresponds to the actually detected movement information 130 in FIG.
0 corresponds to the motion parameter information approximated by the function equation determination unit 5. In FIG. 8, 0 ≦ t <t1
Although the optimal function 1 satisfies the condition of L <T and the error was within the allowable range, at time t = t1, L ≧ T, and the optimal function 2 was selected. Subsequently, L <T at t1 ≦ t <t2, but L ≧ T at t = t2, and the optimal function 3
Is selected. Therefore, the three optimal functions are switched during 0 ≦ t ≦ t3, and the motion information is converted into a function equation / coefficient.
That is, the approximation is made by the function equation registered in the database 6.

On the other hand, on the receiving side, as shown in FIG.
The actual motion of the subject is calculated from the motion parameter 170 and the function equation / coefficient / database 6, and the calculated motion is given to the wire frame model registered in the three-dimensional model 8. The output motion information 190 of the subject is input to the composite image output unit 10 and is a digitized input image signal 90 which is also an output on the transmission side.
Of the composite image signal 2 from the composite image output unit 10
00 is obtained.

The above is the outline of the first embodiment. Next, a specific example will be described. In the motion of the human head model, a certain degree of periodicity is recognized in the rotation of the head such as nodding and swinging, and the vertical movement of the jaw due to opening and closing of the mouth during conversation. Therefore, in order to realize the periodicity, the equation of motion of the vibration system of the equation (1) shown in FIG. 9A is introduced.

The form of the general solution differs depending on the sign of the characteristic equation c ² -4mk of the equation (1). Here, assuming that m = 0.5, a general solution is obtained by determining the magnitude of c ² −2k and 0. (I) When c ² > 2k Equation (2) is obtained as shown in FIG. 9B. (Ii) When c ² = 2k Equation (3) is obtained as shown in FIG. 9 (c). (Iii) When c ² <2k Equation (4) is obtained as shown in FIG. 9D. Therefore, in this case, the function equations are as described in (2) and (3) above.
If only the equation (4) is used, the parameters k and c are determined, and the corresponding function equation is determined on a one-to-one basis. Therefore, here, an analysis / synthesis encoding method using the above-described transmission of the parameters k and c is considered.

Conventionally, parameters are given for each unit of model movement (rotation about the X, Y, and Z axes). Advantages include a high degree of freedom of movement and a simple description. However, the disadvantages are that smooth movement is difficult, and the amount of information for reproducing a model with large movement becomes large.

In this method, a series of model movements are considered together, and a long sequence is considered to be a continuation of these series of movements. For example, when there is a movement as shown in FIG. 8, 0 ≦ t ≦ t ₁ ... Motion ₁ t ₁ ≦ t ≦ t ₂ ... Motion ₂ t ₂ ≦ t ≦ t _3.・ It is divided into three areas of motion3. Should transmission parameters accordingly (k _1, c _1), it is possible to (k _2, c _2), is only (k _3, c _3), greater data compression than the conventional analysis coding.

FIG. 10 shows a motion pattern in a motion model when k and c are changed.
This indicates that many motions can be approximated. Also,
In the case of the damped vibration according to the equation (4) of the above (iii), since both the amplitude A and the period T are determined only by the parameters of k and c, 1, the magnitude of the movement amount... 2. There is an advantage that the speed of movement... (Corresponding to the period T) can be determined at the same time.

Therefore, the features of the present method are as follows. Advantage 1 The amount of transmitted information is smaller than the conventional method. Advantage 2 Smooth expression of model movement is possible. Advantage 3 Since it is determined immediately which of the three equations is used depending on the magnitude relationship between the values of k and c, the description is extremely easy. However, the disadvantages are as follows. Disadvantage 1 An error occurs because the motion must be approximated to the actual motion.

In the above specific example, the case where the functional equations are the equations (2), (3) and (4) and the number of coefficients is k and c is shown. In this example, since the determination can be made only from the values of k and c in the equations (2), (3), and (4), the motion is transmitted by transmitting only the coefficient group (k and c) as a parameter. Could be approximated,
For example, when expressions other than the expressions (2), (3), and (4) are also registered in the database, it is necessary to transmit the function equation and its coefficient group .

It is also conceivable to register a function equation and a fixed coefficient in a database as a set. For example, equation (2) is registered as a function equation for function number 1 and registered as coefficient values with k = 1000 and c = 0.01 fixed. Similarly, the equation (2) is registered as the function equation in the function number 2 and registered as coefficients in which k = 10000 and c = 0.1 are fixed. In this way, by registering a plurality of coefficients approximating a motion close to an expected motion in the same function equation in advance, the parameter to be transmitted only needs to be a function number.

In the above embodiment, the method of selecting and determining a new function equation at t ₁ , t ₂ , and t _{3 as} shown in FIG. 7 and the method of determining the system value of the function have not been described. As an example of this method, B-spli
An approximation technique using a ne curve or an approximation technique using a Bezier curve can be used.

Further, in the above embodiment, FIGS.
Although FIG. 10 describes only the X axis, the same applies to the Y axis and the Z axis. Then, these can be combined to approximate a three-dimensional movement. If 2
If it is sufficient to approximate the movement of only the dimension, only the X-axis and the Y-axis are sufficient. Alternatively, processing may be performed on any two axes. If you are satisfied with approximation of the movement of only one-dimensional, have
What is necessary is just to process about one axis | shaft.

Embodiment 2 FIG. In the first embodiment, since the digitized input image signal is transmitted as it is, no other subject image can be obtained on the receiving side, but the composite image output unit 1 on the receiving side in FIG.
By connecting an image database to the digital image signal 90 and multiplexing the addresses of the image numbers used for the composition into the digital input image signal 90, any image registered in the database can be provided, and the motion of the subject in the input image can be provided. Is detected, a moving image of another subject can be obtained on the receiving side.

Embodiment 3 FIG. In the first embodiment, the image input unit 1 defines a single-eye (single) camera input. However, by using this as a compound-eye (multiple) camera input, the measurement target range is widened and the subject is wider. The movement of the area can be tracked.

Embodiment 4 FIG. In the first embodiment, the three-dimensional model 8 is provided in advance.
By matching the wire frame model in the three-dimensional model 8 with the subject image, the movement in the Z direction (depth direction) can also be detected. However, as shown in FIG. If the image processing method is used from the obtained slit light image 210 and the obtained slit light image 210 is used, the three-dimensional feature point extraction unit 20 can calculate the three-dimensional coordinate information 220 of the feature point on the surface of the target object. Although the set of points in FIG. 12 is the three-dimensional coordinate information 220 of the feature points, it can be seen that the shape of the person's head is well extracted. Therefore, by tracking the three-dimensional coordinate information 220 of the feature points over two frames, the movement of the subject image can be accurately detected as in the first embodiment. As described above, in the fourth embodiment, the input image portion includes:
A moving image transmission apparatus has been described in which a slit light projector and a three-dimensional feature extraction unit are arranged and a three-dimensional structure of an object is detected by an image processing technique.

Embodiment 5 FIG. In the first embodiment, the function equation / coefficient / database is fixed. However, by adding a memory, more data of function equations and coefficients can be registered. This increases the chances that an optimum function for obtaining a closer approximation to the actual motion of the subject image will be given.

Embodiment 6 FIG. In the first embodiment, the case where the function equation / coefficient database is registered in advance is shown. However, the movement of the subject generated between a plurality of frames of the input image is recorded in real time in a self-organizing database or a neural network database. Automatically register and, if the same motion is detected again in the subject, output the motion parameters from the database, omit the real number calculation in the function equation determination unit and the error value threshold processing unit, and increase the speed. The moving image transmission device may be characterized in that

Embodiment 7 FIG. In the first embodiment, the threshold value of the error value threshold processing unit is fixed. However, in order to make the transmission speed of the moving image transmission device of the present invention variable, it is possible to change the threshold value. This makes it possible to adapt to the transmission capacity of the communication path for communication.

In the above description, the case where the present invention is used as a moving image transmitting apparatus has been described. However, since the present invention has a function of extracting three-dimensional information, the present invention is used for detecting and transmitting information such as changes in facial expressions of a person. It goes without saying that you can do it.

[0038]

Since the present invention is configured as described above, it has the following effects.

By matching the registered wireframe model with the subject image in the input image signal, the motion of the subject can be detected over a plurality of frames.

Even when the motion of the subject image is complicated, the motion can be approximated over a plurality of frames by using a function equation prepared in a database in advance and a coefficient group thereof.

According to the above-described effect, transmission of a moving image can always be realized with a small amount of information (low bit rate).

By increasing the number of elements in a plurality of function equations and their coefficient groups, it is possible to flexibly cope with complicated movements of a subject.

In the error value threshold value processing unit, if the approximation error due to the function equation is equal to or larger than the threshold value, it is approximated using another optimal function. High-quality moving images can be obtained on the receiving side.

By making the error value in the error value threshold value processing section variable, it is possible to make the transmission speed of the moving picture transmission apparatus of the present invention variable.

[Brief description of the drawings]

FIG. 1 is a block diagram of a transmitting side showing a first embodiment of the present invention.

FIG. 2 is a block diagram of a receiving side showing the first embodiment of the present invention.

FIG. 3 is a diagram illustrating movement of a subject over two frames.

FIG. 4 is a diagram showing three feature points of a human head to be detected.

FIG. 5 is a diagram of a three-dimensional model on a shoulder.

FIG. 6 is a diagram of a three-dimensional model of a head.

FIG. 7 is a structural diagram relating to data such as a functional equation and its coefficient group.

FIG. 8 is a diagram in which an actual motion of a subject and a motion approximated by an optimal function are two-dimensionally displayed on the same time axis.

FIG. 9 is a diagram showing an example of a specific function equation according to the present invention.

FIG. 10 is a diagram showing an example of a specific divisor group of the present invention.

FIG. 11 is a block diagram of a preceding stage on a transmitting side, showing a fourth embodiment of the present invention.

FIG. 12 is a diagram of feature points on an object extracted by the slit light projection method according to the fourth embodiment of the present invention.

FIG. 13 is a block diagram showing a conventional moving image transmission device.

[Explanation of symbols]

DESCRIPTION OF SYMBOLS 1 Image input part 2 3D model matching part 3 Frame memory 4 Motion detection part 5 Function equation determination part 6 Function equation / coefficient / database 7 Error value threshold processing part 8 3D model 9 Motion model creation part 10 Synthetic image output part 11 Position of subject in previous frame 12 Position of subject in current frame 13 Center of left eye 14 Center of right eye 15 Top of nose 20 3D feature point extraction unit 30 Slit light projector 50 Transmission unit 51 Reception unit 52 Face 3 Dimensional model 90 Digitized input image signal 100 Connection information / vertex information of 3D model 110 Input image information after wireframe matching 120 Input image information after wireframe matching one frame before 130 Motion of subject after motion detection Information 140 Information on function equations and their coefficients 150 Approximation by function equations Motion parameter information 160 Flag 170 Motion parameter (number of function equation and coefficient group) 190 Motion information 200 Synthetic image signal 210 Input image obtained by projecting slit light on target object 220 Extracted three-dimensional feature point 500 Input image 510 Analysis Parameter 520 Wireframe information 530 Output image

──────────────────────────────────────────────────続 き Continuation of front page (56) References JP-A-62-130087 (JP, A) IEICE Technical Report, NL U90-28, PRU90-28 1-8 (High-precision estimation of head and face motion information in intelligent image coding) (58) Fields investigated (Int. Cl. ⁶ , DB name) H04N ^7/ 24-7/68

Claims (5)

(57) [Claims] 1. The method according to claim 1, wherein each of the three-dimensional directions has a time axis.
A plurality of functional equations correspondingly representing the movement of the subject;
Function equation / coefficient / database with multiple coefficient groups registered
Graphics and the movement of the subject detected in the input image, as the function side
A function equation to be approximated over a plurality of frames is determined by using a function equation registered in an equation / coefficient / database. A function equation determination unit, the motion of the detected subject and the motion approximated by the function equation determination unit It calculates an error value between performs size determination of the error value and the threshold value, a function equation as information of a function equation that gives a more approximate motion
And a error value threshold processing unit for outputting a coefficient group, using the function equation is determined by the function equation determining unit
To approximate the movement of the subject, and
By determining the error value, the motion of the subject is
Detects cases where approximation is no longer possible with numerical equations, and
Restart the number equation determiner to determine a new function equation
And approximate the subject's movement by the determined function equation.
A moving image transmission device capable of transmitting a moving image with a small amount of information . 2. The moving picture transmission apparatus according to claim 1, wherein the function equation determining unit is configured to determine the function equation / coefficient / data.
The motion is approximated using a function equation and a coefficient group registered in the database, and the error value threshold processing unit outputs function information that gives a more approximated motion, and outputs the function equation / coefficient / data.
Database as the transmitting side and the moving image transmitting apparatus as the receiving side.
Share the same functional equations and coefficient groups with moving image transmission equipment
By doing so, it becomes information indicating the function equation and coefficient group.
The function number from the moving image transmission device on the transmitting side to the receiving side.
Output to a moving image transmission device
A moving image transmission device capable of transmitting a moving image with information . 3. A moving image transmitting apparatus, comprising: a model creating section for reproducing a motion of a subject based on information output from the moving image transmitting apparatus according to claim 1. 4. A moving image transmission method having the following steps: (a) corresponding to a time axis in each of three-dimensional directions;
Multiple function equations and multiple relationships
The several groups, registration step of registering as a database as a function number information you approximate the motion, (b) an image input step of inputting the motion of an object, the detection step of detecting a motion of a subject from (c) an input image, (D) selecting and determining function information approximating the detected motion of the subject from the database and outputting the determined function information ; (e) using the detected motion of the subject and the function information selected in the determining step. An error value threshold processing step of restarting the determination step based on a difference from the approximated motion, and using the functional equation determined in the determination step,
Is approximated, and a new
When the information of the function equation is output,
Using the function equation determined in the determination step,
Approximate motion to transmit video with less information
Video transmission method that enables 5. A moving image transmission method, comprising the step of: inputting function information output by the moving image transmission method according to claim 4, and generating a model for reconstructing the movement of a subject.