JP3341575B2 - Image synthesis device - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image synthesizing apparatus for synthesizing an image at a new viewpoint using an existing image.

[0002]

2. Description of the Related Art FIGS. 25, 26 and 27 show, for example, Akihiro Katayama, Koichira Tanaka, Takahiro Oshino, and Hideyuki Tamura, "3D Image Conference with Viewpoint Tracking Type Stereoscopic Image Display by Interpolation Processing of Multiview Image Data". 94 Proceedings, p
p. FIG. 7B is an explanatory diagram showing the operation of the conventional image synthesizing apparatus shown in FIGS. 7-12 and 1994. 1 is an input image, 1a
Is an input image having the intersection 10 as a viewpoint, 2 is a viewpoint at which the input image 1 is captured, 3 is an optical axis at which the input image 1 is captured, 4 is a newly synthesized image, 5 is a viewpoint of a newly synthesized image 4, 6
Is the optical axis of the image 4 to be newly synthesized, 7 is the projection plane of the composite image 4, 8 is an area on the projection plane 7, 9 is a line of sight connecting the viewpoint 5 and the area 8, and 10 is a line of sight 9 and the x-axis. Intersection of 12, 1
2a is the projection plane of the input image 1 or the input image 1a, 13 is the i-th vertical component of the input image 1a, 14 is the i-th vertical component of the composite image 4, 15 is the object, 16 (FIG.
6) is a second object whose distance from the x-axis is different from that of the object 15, and 54 (FIG. 27) is an area generated by a specific input image 1 in the composite image 4.

In a conventional image synthesizing apparatus, as shown in FIG. 25, a multi-viewpoint input image 1 (Ix (i, j) taken with the viewpoint 2 placed on the x-axis and the optical axis 3 parallel to the z-axis. )), The image 4 in which the optical axis 6 at the new viewpoint 5 is parallel to the z-axis.
(J (i, j)) are synthesized as follows. For the new viewpoint 5, for each i of the image 4, a line-of-sight line 9 on the zx plane connecting the region 8 on the projection plane 7 corresponding to the component 14 in the vertical direction of the image 4 and the viewpoint 5 is obtained. An intersection 10 between the line of sight 9 and the x-axis is determined. Let the x coordinate of this intersection 10 be x
(I). The line-of-sight line 9 passes through a region corresponding to the i-th vertical component 13 of the projection plane 12a of the input image 1a (Ix (i) (i, j)) with the intersection 10 as a viewpoint, and the i-th line of the input image 1a The i-th vertical component 13 of the input image 1a and the i-th vertical component 14 of the image 4 have the same line of sight, and represent the same object 15. The composite image 4 is generated by using the vertical component 14 of. At this time, usually the viewpoint 5
The object 15 must be drawn smaller in the composite image 4 as the distance from the object 15 increases. To achieve this effect,
Assuming that the z coordinate of the object 15 is T, the vertical component 13 is enlarged or reduced T / (T + z) times by the z coordinate value of the new viewpoint 5. That is, for each vertical component 14 of the composite image 4,

[0004]

(Equation 1)

[0005] A composite image 4 is generated as described above.
However, in FIG. 25, since the direction of the z-axis is opposite to that of the above document, the sign of the above-mentioned magnification z is also opposite to that shown in the above document.

[0006]

The conventional image synthesizing apparatus as described above has a problem that the Z coordinate value T of the object 15 in the input image, that is, the position of the object 15 must be known in advance. Was. Also, FIG.
As shown in FIG. 27, when the second object 16 having a distance different from T is also present as shown in FIG.
The target object 16 is distorted in the vertical direction on the composite image 4, and there is a problem that a problem occurs when the user tries to pay attention to the target object 16. Furthermore, when the vertical component 13 is used from the plurality of input images 1 to generate the composite image 4, as shown in FIG. 27, the luminance differs even between the input images depending on the illuminance condition at the time of shooting. ,
For example, an input image I from the viewpoint of x (i) shown in FIG.
If x (i) (i, j) is underexposed, the area 54 of the composite image 4 that uses the input image Ix (i) (i, j) will be dark. When a plurality of input images are used as described above, there is a problem that luminance discontinuity occurs in the composite image 4 and the image becomes unnatural. Also, when a large number of input images 1 are used, it is difficult to store all of them on the main storage device of the computer. There is a problem that the processing time increases each time an image is referenced.

The present invention has been made in order to solve such a problem, and does not require a process of obtaining a distance to the object 15 in advance, and is capable of coping with a plurality of objects having different distances. The purpose is to obtain a synthesizer. It is another object of the present invention to provide an image synthesizing apparatus that obtains a synthesized image having no discontinuity in luminance. It is another object of the present invention to provide an image synthesizing apparatus that can obtain a synthesized image at high speed even when a large number of input images are used.

[0008]

In the image synthesizing apparatus according to the first aspect of the present invention, the distance from the viewpoint to the object to be focused can be arbitrarily adjusted when a synthetic image is created.

In the image synthesizing apparatus according to the second aspect of the present invention, the distance from the viewpoint to the object to be focused can be arbitrarily adjusted when the synthesized image is created, and the image is displayed in the vertical direction of a pixel having a width of 1. The processing is performed by dividing into components.

In the image synthesizing apparatus according to the third aspect of the present invention, the distance from the viewpoint to the object to be focused can be arbitrarily adjusted when the synthesized image is created, and the partial image of the input image is converted to the synthesized image. When transferring to a partial image, a plurality of partial images of the input image are used.

In the image synthesizing apparatus according to a fourth aspect of the present invention, the image is divided into a certain width in the vertical direction and processed.

In the image synthesizing apparatus according to a fifth aspect of the present invention, the distance from the viewpoint to the object to be focused can be interactively input at any time when creating the composite image, and the distance and the use of the composite image can be used. The magnification of enlargement or reduction is changed using the distance between the viewpoints of the input image to be enlarged.

In the image synthesizing apparatus according to the sixth aspect of the present invention, the distance from the viewpoint to the object to be focused can be interactively input at any time when creating a composite image, and the distance between each partial image of the composite image is A different value can be set, and the magnification for enlargement or reduction is changed using this distance and the distance between the viewpoints of the composite image and the input image to be used.

In the image synthesizing apparatus according to the present invention, when calculating the enlargement / reduction ratio, the visual line is obtained for each of the partial image of the composite image and the partial image of the input image to be transferred thereto, and the magnitude of the visual line direction is obtained. The inner product of the vector of No. 1 and the position vector of the viewpoint is calculated, and the distance between the viewpoint of the synthesized image and the viewpoint of the input image is obtained from the difference between the two.

In the image synthesizing apparatus according to the present invention, in the synthesized image generated by transferring the partial image of the input image, the pixels of the other input image are divided at the boundary of the area divided for each transferred input image. The values are mixed.

In the image synthesizing apparatus according to the ninth aspect, the mixing process of the pixel values at the boundary of the composite image is sequentially executed from one end of the composite image, and the mixing process is performed only on one side of the boundary. In this way, the range having unprocessed boundaries is prevented from being mixed.

In the image synthesizing apparatus according to the tenth aspect, when the viewpoint of the synthesized image is moved, a partial image of the input image which is considered to be used for transfer after the movement is preferentially stored in a high-speed storage device. It is something to keep.

In the image synthesizing apparatus according to the eleventh aspect of the present invention, a partial image group in which a plurality of partial images of an input image are put together is stored as a unit.

In the image synthesizing apparatus according to the twelfth aspect of the present invention, when the viewpoint of the composite image is moved, the line of sight of the partial image of the input image and the current composition are used as a measure of whether or not the viewpoint is used for transfer after the movement. The distance from the viewpoint of the image is used.

[0020]

BEST MODE FOR CARRYING OUT THE INVENTION

Embodiment 1 FIG. FIG. 1 is a block diagram showing a first embodiment of the present invention. Reference numeral 17 denotes an image input device (image input device) which is a device for inputting an image, and reference numeral 18 denotes a viewpoint position of an image input from the image input device. Viewpoint information input device (input means) which is a means for inputting information on the direction of the optical axis and the angle of view, 19
Is an input image processing apparatus (image dividing means) which is a means for decomposing an input image into partial images and processing the image; Gaze determining device (gaze calculation means), 21 is a storage device for storing the image, viewpoint information, and gaze information used for structuring, 22 designates a new viewpoint position and optical axis direction for image synthesis A point-of-view instruction device (input unit) 23 is a focus adjusting device (adjustment unit) that adjusts the focal position of the composite image 4.
Reference numeral 4 denotes a combined image generation device (transfer unit) which is a unit for combining an image having the viewpoint position and the optical axis direction input from the viewpoint designating device 22. Reference numeral 25 denotes a search device (transfer device) for searching for a partial image using the visual line information. Means, 26 is the composite image generation device 2
4 is a display device that is a means for outputting an image synthesized by the image forming apparatus 4.

FIGS. 2 and 3 are flowcharts showing the operation of the first embodiment of the present invention. FIGS. 4, 5, 6 and 7 are explanatory diagrams showing the operation of the first embodiment of the present invention. 27 is an input partial image obtained by decomposing the input image 1, 28 is an input partial image 2 on the projection plane 12
7, 29 is a line-of-sight vector of the input partial image 27, 30 is the origin, 31 is a perpendicular foot lowered on a straight line on which the line-of-sight vector extends from the origin 30, 32 is an image 4 to be synthesized
33 is a partial image 32 in the projection plane 7
Is a line-of-sight vector of the partial image 32,
D is the distance from the viewpoint 5 of the composite image 4 to the object 15;
5 is an axis perpendicular to the zx plane and passes through the viewpoint 5 of the composite image 4,
It is a cylindrical surface having a radius of D.

In the image synthesizing device configured as described above, instead of the z coordinate value T determined by the object,
The distance D from the viewpoint 5 of the region to be accurately represented on the composite image 4 is used. Further, when the distance D is unknown, an arbitrary temporary value is used as the value of the distance D in the process of repeatedly synthesizing the composite image 4 and the distance D is appropriately determined interactively while referring to the composite image 4. While changing, an accurate composite image is finally obtained. The distance D is information that is difficult to obtain as input image information, and if the user sticks to obtaining this information, image synthesis becomes basically difficult. First, the corresponding viewpoint position 2 and a variable expressing the line of sight 29 are associated with a partial image 27 which is a vertical component of the input image 1.
This operation will be described with reference to the flowchart of FIG. This processing is mainly executed by the input image processing device 19. In step ST1, the number N of frames of the input image
Is set, and n indicating the number of the input image is initialized to 1. In step ST2, the n-th input image 1 is input by the image input device 17, and this is input to In (i, j),
i = 1,. . , Mx, j = 1,. . , My at the same time, the position of the viewpoint 2 (x-coordinate value is x, z-coordinate value is z) and the direction と す る with respect to the z-axis of the optical axis 3, The angle of view φ of the camera is input. Image input by the image input device 17 can be realized by capturing an image with a video camera, reading image data stored in a magnetic disk device, or the like. The input of the viewpoint information by the viewpoint information input device 18 is performed, for example, by attaching a position sensor to the camera when using a video camera, and by measuring the position information, and when using a magnetic disk device, storing the viewpoint information together with the image data. , And can be realized by reading them out. In step ST3, the horizontal coordinate io of the input image 1 is initialized to 1. Step ST
In step 4, the input image processor 19 extracts the input partial image 27 from the image 1. This is the io-th vertical column. In step ST <b> 5, the line-of-sight determination device 20 obtains a line-of-sight vector 29 from the viewpoint 2 at which the image 1 was captured to a portion 28 on the projection plane 12 corresponding to the input partial image 27. As shown in FIG. 4, the projection of the line of sight onto each pixel of the input partial image 27 on the zx plane is a single vector on the zx plane. This vector is referred to as a line-of-sight vector 29. To specify the vector, the directions θ, ρ, and ε are

[0023]

(Equation 2)

Determined as described above. Note that θ is 0 ≦ θ <
2π. The direction θ is the direction of the line-of-sight vector 29,
ρ is obtained by adding a sign to the distance between the origin 30 and the straight line on which the line-of-sight vector 29 is placed. ε is the distance from the perpendicular foot 31 lowered from the origin 30 on the straight line on which the line-of-sight vector 29 rides to the viewpoint 2 measured in the direction of the line-of-sight vector 29. Further, as shown in Expression (4), ε is an inner product of the unit vector in the line-of-sight direction and the position coordinates of the viewpoint 2. With this representation, it is possible to correspond to an arbitrary line-of-sight vector 29 on the zx plane. In step ST6, the input partial image 27 and the variables are stored in the storage device 21 in association with each other. That is, the variables are stored in arrays P, Q, and R indicating the variables of the io-th input partial image 27 of the n-th input image 1. Step ST
At 7, io is increased by one. In step ST8, it is determined whether or not io is less than or equal to Mx. If it is less than or equal to Mx, the process returns to step ST4; otherwise, the process proceeds to step ST9. In step ST9, n is increased by one. In step ST10, n
Is smaller than or equal to N. If it is smaller than N, the process returns to step ST2. Otherwise, the image input and storage process ends.

Next, the operation of synthesizing an image having a new viewpoint and an optical axis direction will be described with reference to the flowchart of FIG. Let the composite image 4 be J (i, j). As with the input image 1, the partial image 3 corresponding to the vertical component
A line-of-sight vector 34 can be determined by the corresponding area 33 on the projection plane 7 for each of the two. If there is an input partial image 27 having a line-of-sight vector 29 that is on the same straight line as the line-of-sight vector 34, has the same direction, and has a near viewpoint position, the partial image 27 is expected to be the partial image 32 of the composite image 4. Pattern. Thus, the composite image 4 is generated by searching for the partial image 27 of the input image 1 having the line-of-sight vector that can be regarded as the same for each of the partial images 32 of the composite image 4 and transferring the contents thereof. In the process of repeating the synthesizing process, the distance D to be accurately expressed from the position of the viewpoint 5 of the synthesized image 4 is appropriately changed interactively while referring to the synthesized image 4 to obtain the target object 15 to be noted.
Go to the optimal value for Adjustment of the distance D is exactly the same operation as focusing the camera. By having this function, even when the position of the object 15 is not known in advance, it is possible to generate the composite image 4 that accurately represents the object 15.

In step ST20, the viewpoint designating device 22
(The x coordinate value is x, z)
The coordinate value is assumed to be z) and the direction の of the optical axis 6 is input and sent to the composite image generation device 24. This input is realized by, for example, a mouse or a keyboard. The following is mainly executed by the composite image generation device 22. In step ST21, the focus adjustment device 23 inputs a distance D for focusing from the position of the viewpoint 5 of the composite image 4. This D is shown in FIG.
In the course of the repetition of the processing shown in the flowchart of FIG. This focus adjustment can be realized, for example, by directly inputting the value of D using a keyboard, or by instructing an increase or decrease from the previous D value using a mouse, a keyboard, or a joystick. If there is no input, the operation may be performed without waiting for the input, using the previous value of D or the preset initial value as the value of D.

The following is mainly executed by the composite image generating device 24. In step ST22, a variable io indicating the processing position of the image is initialized to 1. In step ST23,
The line-of-sight determination device 20 obtains a line-of-sight vector 34 from the viewpoint 5 of the composite image 4 to the partial image 32. this is,
The same operation as in step ST5 is performed, and the gaze vector 34
Θ, ρ, and ε are obtained. In step ST24, the search device 25 searches for the best matching input partial image using (θ, ρ, ε). This is, for example, the array P,
Using the values stored in Q and R,

[0028]

(Equation 3)

The minimum n and i are obtained, and these are calculated as n *,
The i * -th partial image of the n * -th input image is obtained as i *. Here, w1, w2, and w3 are parameters having positive or zero values indicating weights. When the specific w is set to 0, it is not required to match the variable corresponding to the input image between the input image and the composite image, and the input image can be easily selected. Here, it is assumed that w3 is set to 0, and it is not required to match ε, and an input image that is on the same straight line as the line-of-sight vector 34 of the composite image 4 but has a viewpoint different from the viewpoint 5 can be used. In step ST25, the i * -th partial image of the n * -th input image stored in the storage device 21 is read and transferred to the partial image 32 of the composite image 4. At this time, using D,

[0030]

(Equation 4)

As described above, the image is transferred after being enlarged or reduced. Here, Δε is a signed distance from the viewpoint 5 of the composite image 4 to the viewpoint 2 of the input image 1 on a straight line on which both the gaze vectors are placed. Α and β are coefficients for correcting the magnification of the object obliquely in the direction of the optical axis in consideration of the perspective projection. In the peripheral part of the image, even if there is an object at the same distance from the viewpoint as compared with the central part, the object is largely expressed on the image.
In order to perform the expression with the correct size by performing the enlargement / reduction according to the distance from the viewpoint, the change in the size due to the image position is corrected. α means conversion to an input image such that objects at the same distance have the same size, that is, a cylindrical surface as a projection surface, and β means a conversion to a cylindrical surface with the viewpoint 5 as the center. This represents conversion of the composite image 4 obtained as a projected image into an image projected on a projection plane 7 which is a plane. That is, when performing transfer from an input image having a different optical axis angle from that of the composite image 4, the image is first converted into a projection on a cylindrical surface, and then a conversion corresponding to the optical axis angle on the composite image is performed to perform accurate enlargement. The reduction is performed. When a cylindrical surface having the viewpoint as the central axis is used as the projection surface, the size of the object in the image is inversely proportional to the distance from the viewpoint. As shown in FIG. 5, the object 15 at the distance D is at a distance of D−Δε from the viewpoint 2 of the input image 1. Therefore, by enlarging or reducing the partial image 27 of the input image 1 by (D−Δε) / D times as shown in the equation (6), the object 15 at the distance D becomes the correct size in the composite image 4. Become.

In step ST26, one is added to io.
In step ST27, it is determined whether or not io is equal to or less than Mx, that is, whether or not there is an unprocessed partial image 32. If there is, the process returns to step ST23;
Proceed to 8. In step ST28, the obtained composite image 4
Is displayed on the display device 26, and the image synthesis processing at the instructed viewpoint 5 is completed.

By the above-described processing, a composite image in which the distance from the viewpoint 5 shown in FIG. 6 is D, that is, the object 15 on the cylindrical surface 35 having a radius D centered on the viewpoint 5 is accurately represented. Obtainable. This D can be obtained interactively during the processing, and does not need to be precisely determined in advance. Further, unlike the case where the z-coordinate value is used, the composition can be performed by the same processing regardless of the direction of the optical axis 6 of the composite image 4. FIG. 7 shows an example of the operation. An operation of adjusting the distance D, whose value was D1 in accordance with the target object 15 initially, to D2 in order to focus on the second target object 16 is shown.

Embodiment 2 A block diagram showing a second embodiment of the present invention is the same as FIG. This embodiment aims at speeding up image synthesis processing.
FIG. 8 is an explanatory diagram showing the operation of the second embodiment of the present invention. FIG. 9 is a flowchart showing the operation of the second embodiment of the present invention. 36 is a partial image of pixels of width L obtained by dividing the synthesized image 4, 37 is a corresponding region on the projection plane 7 corresponding to the partial image 36, 38 is an input partial image whose view vector matches the partial image 36, 39 is a partial image This is a region near the input partial image 38 used for generating the image 36.

In the image synthesizing apparatus according to the second embodiment of the present invention, as shown in FIG. 8, the synthesized image 4 is divided into partial images 36 each having a width L composed of a plurality of vertical lines. The synthesizing process is performed. In the first embodiment, the synthesis process is performed line by line in the vertical direction, whereas the synthesis process is performed line by line. Therefore, when L is increased, the trouble of searching for and transferring a suitable image is reduced and the processing speed is reduced. improves.

In the image synthesizing apparatus configured as described above, first, as in the first embodiment, the image 1 and the accompanying information are input and stored in the storage device 21. This operation is the same as in the flowchart of FIG.

Next, the operation of synthesizing an image having a new viewpoint and a new optical axis direction will be described with reference to the flowchart of FIG. Let the composite image 4 be J (i, j). The gaze vector 34 is determined by the corresponding area 37 on the projection plane 7 for each partial image 36 having the width L. Using an input partial image 38 having a line-of-sight vector 29 in the same direction as the line-of-sight vector 34 and having a line-of-sight vector 29 in the same direction, and a partial image in the vicinity 39 thereof,
A partial image 36 is generated.

In step ST30, step ST20
In the same manner as described above, the viewpoint 5 position (x coordinate value is x and z coordinate value is z) and the direction 光 of the optical axis 6 of the image 4 to be combined by the viewpoint designating device 22 are input and sent to the combined image generating device 24. . This input is performed by, for example, a mouse or a keyboard. The following is mainly executed by the composite image generation device 24. In step ST31, the focus adjustment device 23 inputs a distance D for focusing from the position of the viewpoint 5 of the composite image 4. This is the same as the operation in step ST21. In step ST32, a variable io indicating the processing position of the image is initialized to 1. In step ST33, the line-of-sight determination device 20 starts (io-
1) Obtain a line-of-sight vector 34 from × L + 1 to a partial image 36 corresponding to the io × L-th vertical component. This corresponds to the center of the partial image 36, that is, ｛io × L− (L
−1) / 2｝ th vertical component I (io × L− (L−
As the line-of-sight vector for 1) / 2, j), θ and ρ of the line-of-sight vector 34 by the same operation as in step ST5.
And ε. When L is an even number, io × L−
A fractional part of (L-1) / 2 is rounded up or down and a corresponding vertical component is used. In step ST34,
The search device 25 searches for the best matching input partial image using (θ, ρ, ε). This is, for example, by the values stored in the arrays P, Q, R,

[0039]

(Equation 5)

The minimum n * and i * are obtained, and are set as the i * -th partial image of the n * -th image. Step ST35
Then, the i * -th partial image 38 of the n * -th input image stored in the storage device 21 and the partial image of the vicinity 39 thereof are

[0041]

(Equation 6)

As described above, the image is transferred to the partial image 36 of the composite image 4. At this time, enlargement / reduction is performed by focusing on the distance D from the viewpoint 5. Since the size of the object in the image is inversely proportional to the distance from the viewpoint, the object at the distance D becomes the correct size in the composite image 4 by this scaling. In the second embodiment, different from the first embodiment, scaling is performed at the same magnification not only in the vertical direction but also in the horizontal direction. In step ST36, 1 is added to io. In step ST37, it is determined whether (io-1) × L + 1 is equal to or smaller than Mx, that is, whether there is an unprocessed partial image 36. If there is, the process returns to step ST33, and if not, the process proceeds to step ST38. In step ST38, the obtained synthesized image 4 is displayed on the display device 26, and the image synthesis processing at the instructed viewpoint 5 is completed.

By the above-described processing, a composite image accurately representing the object at an arbitrary distance D can be obtained.

Embodiment 3 FIG. FIG. 10 is a block diagram showing a third embodiment of the present invention.
Image smoothing device (smoothing means) that eliminates discontinuity in luminance by mixing both pixel values at the middle boundary
It is. FIGS. 11 and 12 are flowcharts showing the operation of the third embodiment of the present invention. FIGS. 13, 14 and 15 are explanatory diagrams showing the operation of the third embodiment of the present invention. 41 is an area of the composite image 4 generated by the same partial image of the input image 1, 42 is an area adjacent to the area 41 and generated by a partial image of the input image 1 different from the area 41, and 43 is the composite image 4 Middle area 4
A boundary between 1 and the area 42, 44 is a smoothed area in the area 42 on which a smoothing process is performed, and 45 is a smoothed synthesized image obtained by performing the processing described below on the synthesized image 4. 46 is a region adjacent to the region 42 and generated by a different partial image of the input image 1, and 47 is a boundary between the region 42 and the region 46 in the composite image 4.

In the image synthesizing apparatus according to the third embodiment of the present invention, the synthesized image 4 is subjected to a smoothing process. FIG.
As shown in the figure, a boundary 43 between an area 41 generated by the same input image 1 of the composite image 4 and an area 42 generated by a different input image 1 differs due to a difference in photographing conditions of both input images 1. This causes a luminance discontinuity. In order to smooth this, in the smoothed area 44 having the width K facing the boundary 43 of the area 41, the pattern of the input image used for the area 42 is gradually mixed. As a result, discontinuity in luminance is eliminated, and the naturalness of the composite image 4 is improved. As in the first embodiment, first, the image 1 and accompanying information are input and stored in the storage device 21. This operation is described in FIG.
This is the same as the flowchart of FIG.

Next, the operation of synthesizing an image having a new viewpoint and a new optical axis direction will be described with reference to the flowchart of FIG.

Step ST2 in the flowchart of FIG.
Steps 0 to ST28 are the same as the operations of the first embodiment shown in the flowchart of FIG. In step ST29, the number n * of the input image 1 used for generating the partial image 32 of the composite image 4 is stored in the array V, and the number i * of the input partial image 27 is stored in the array U. In step ST40, the smoothing process of the composite image 4 is executed by the image smoothing device 40. FIG. 12 shows the details of step ST40.

In step ST41 of FIG. 12, a variable io indicating the processing position of the image is initialized to 2, and a variable H indicating the number of boundaries is initialized to 0. In step ST42, for the io-th and io-1st partial images of the composite image 4,
It is determined whether or not the used input images are the same, and if they are the same, the process proceeds to step ST44; otherwise, the process proceeds to step ST43.
Proceed to. Not being the same means that the io-th and io-1
Since the second partial image belongs to different regions 41 and 42 and a boundary 43 exists between them, a luminance discontinuity occurs due to a luminance difference between input images. Step S
At T43, H is incremented by 1 and io is stored in the array W indicating the boundary. In step ST44, io is increased by one. In step ST45, it is determined whether or not io is equal to or less than Mx.
If so, the process returns to step ST42; otherwise, the process proceeds to step ST46.

In the following, a smoothing process is performed on the detected boundary portion. As shown in FIG. 13, in the smoothed area 44 having a width K facing the boundary 43 of the area 41 of the composite image 4, the pattern of the input image used for the area 42 is gradually mixed. Even if the width of the area 41 is smaller than K, a smooth image is easily obtained. In step ST46, W [H
+1] is stored as Mx + 1, and h indicating a boundary number
Is initialized to 1. In step ST47, it is determined whether h is equal to or less than H, and if so, the process proceeds to step ST48,
Otherwise, the process of step ST40 ends, and the process returns. In step ST48, a variable k indicating the processing position of the image is initialized to 1. In step ST49, the boundary 43
Then, it is determined whether or not W [h] -k indicating the region 41 side by k pixels is 0 or less. If so, the process goes out of the image and the processing is omitted, and if not, the process proceeds to step ST50. In step ST50, the input image used for the W [h] -th partial image is selected from the W
[H] -Select the input partial image that is most suitable for the k-th partial image. This is, for example,

[0050]

(Equation 7)

The width of the region (W [h + 1] -W
[H]) of the used input partial image (U [W
It is extrapolated using the ratio of [h + 1] -1] -U [W [h]] + 1). In step ST51, the i′-th input partial image 27 of the V [W [h]]-th input image 1 used for the W [h] -th partial image of the composite image 4 obtained by Expression (18) is obtained. , In the W [h] -kth partial image of the composite image 4,

[0052]

(Equation 8)

As shown in FIG. This means that, as shown in FIG. 13, the image used in the area 42 is mixed with the smoothed area 44 of the area 41 by linearly decreasing the ratio according to the distance from the boundary 43. . Step ST
At 52, k is increased by one. In step ST53, k
Is determined to be equal to or smaller than a predetermined smoothing region width K, and if so, the process returns to step ST49; otherwise, the process proceeds to step ST54. In step ST54, h
Is incremented by 1, and the process returns to step ST47 to perform processing on the next boundary.

By the above processing, a synthesized image without luminance discontinuity can be obtained. By sequentially executing the processing in the i direction as described above, smoothing can be performed without any trouble even when the width is smaller than K as in the region 42 shown in FIG. That is, first, the smoothing process is performed on the boundary 43, and then the smoothing process is performed on the boundary 47. In the process on the boundary 43, since the smoothed region 44 does not include the boundary 47, After the above processing, there is no luminance discontinuity in the smoothed area 44. In the subsequent processing for the boundary 47, the boundary 43 included in the target smoothed region 44 has already had its luminance discontinuity eliminated, so that the above-described smoothing processing may cause a new luminance discontinuity. There is no. Also, usually, the boundary 4 of the composite image 4
In 3, the object 15 located at the position of the distance D is drawn at the correct position with the correct size, so that the image does not shift. However, the object 16 is located at a distance other than the focal position D.
If there is, the image is shifted at the boundary 43 of the composite image 4 depending on the viewpoint position of the used input image. Even in such a case, as shown in FIG. 15, if the above-described smoothing process of the boundary portion is performed, the shift is blurred and the unnaturalness can be reduced.

In the above embodiment, the discontinuity of the luminance is eliminated by mixing the images. However, the luminance of each vertical line of the composite image is successively compared, and the luminance change at a substantially constant ratio between the adjacent vertical lines over the entire length of the line. In the case where there is, it is considered that the cause is caused by a difference in photographing conditions, and the discontinuity of the luminance can also be eliminated by a method of forcibly matching the luminance before and after the boundary. The above constant ratio may be treated as a constant difference when the luminance expression is logarithmic.

Embodiment 4 FIG. 16 is a block diagram showing a fourth embodiment of the present invention.
1 (first storage unit), a secondary storage unit (second storage unit) for storing the partial image 26 of the input image 1 that cannot be stored, 49 is a storage for performing storage management of the partial image 26 of the input image 1 It is a control device (storage control means). FIG. 17, FIG. 18, and FIG. 19 are flowcharts showing the operation of the fourth embodiment of the present invention. FIG. 20 is an explanatory diagram showing the operation of the fourth embodiment of the present invention. 50 is a straight line on which the line-of-sight vector 34 rides, 51 is a line-of-sight vector of the partial image 27 of the input image 1 stored in the storage device 21, 52 is a straight line on which the line-of-sight vector 51 rides, and 53 is an input stored in the storage device 21. It is a viewpoint of the partial image 27 of the image 1.

In the image synthesizing apparatus according to the fourth embodiment of the present invention, the generation speed of the synthesized image 4 is increased by controlling the storage location of the partial image 27 of the input image 1. When the number of frames of the input image 1 increases, all of the partial images cannot be stored in the storage device 21 including a main storage device or the like that can execute reading and writing at high speed. Therefore, a part of the data is saved to a large-capacity secondary storage device 48 constituted by a disk device of a computer. At this time, depending on the selection of the partial image to be saved, the partial image saved in the secondary storage device 48 is read out every time the viewpoint 5 of the composite image 4 moves slightly.
It takes a lot of time to generate the composite image 4. In the fourth embodiment of the present invention, the storage controller 4
In step 9, a partial image of the input image to be saved is selected in consideration of the position of the viewpoint 5 of the composite image 4. Thus, the viewpoint 5
In the generation of a new composite image 4 when the image has moved, many of the partial images 27 of the input image 1 that are required
, And the number of times of reference to the secondary storage device 48 is reduced to speed up the processing.

In the fourth embodiment of the present invention,
As in the first embodiment, first, an image 1 and accompanying information are input and stored in the storage device 21 or the secondary storage device 48. This operation will be described with reference to the flowchart of FIG. Steps ST1 to ST10 are the same as the operations of the first embodiment shown in the flowchart of FIG. However, the input image 1 in step ST2
It is assumed that the input image 1 is input by a video camera or the like, or an image stored in the secondary storage device 48 is read. Step ST
At 60, the storage control device 49 stores n in the storage device 21.
It is determined whether there is an area for newly storing the io-th partial image 27 of the input image 1.
Proceed to T61, otherwise proceed to step ST62. This determination is made, for example, by previously dividing the storage capacity of the storage device 21 by the size of the partial image 27 (vertically My pixels) of the input image 1 to obtain the storable number of partial images, This is performed by comparing the stored number. In step ST61, the io of the n-th input image 1
The third partial image 27 is stored in the storage device 21. In step ST62, the io-th partial image 27 of the n-th input image 1 is stored in the secondary storage device. By the above processing, the partial image of the input image 1 that cannot be stored in the storage device 21 is stored in the secondary storage device 48.

Next, the operation of synthesizing an image having a new viewpoint and a new optical axis direction will be described with reference to the flowchart of FIG. Step ST in the flowchart of FIG.
Steps 20 to ST23, and steps ST26 to ST28 and step ST40 are the same as the operations of the third embodiment shown in the flowchart of FIG. Step ST7
At 0, a partial image 27 of the input image 1 is searched in the same manner as in step ST24, but without using the variable ε, for example,

[0060]

(Equation 9)

Then, n * and i * that minimize the value are obtained, and are set as the i * -th partial image of the n * -th image. Step ST71
Then, as in step ST25, the n * -th input image 1
Is read out and transferred to the partial image 32 of the composite image 4. At this time, the storage controller 49 executes the control of the storage device 21 and the secondary storage device 48 with the reading of the partial image 27 of the input image 1. The details of step ST71 will be described with reference to the flowchart shown in FIG.

In step ST80 of FIG. 19, it is determined whether or not the i * -th partial image 27 of the n * -th input image 1 to be read exists in the storage device 21. If there is, the process proceeds to step ST91, otherwise. If the secondary storage device 48
Is read in step ST81 and subsequent steps. This determination is made, for example, by preparing one variable corresponding to each partial image 27, storing or updating the content of the variable in the storage device 21 and referring to the content. In step ST81, n is stored in the storage device 21.
It is determined whether there is an area for newly storing the i * -th partial image 27 of the * -th input image 1, and if there is, the process proceeds to step ST90; otherwise, the process proceeds to step ST82.

In step ST82 and subsequent steps, the storage device 21
And the secondary storage device 48 exchanges the partial images 27. That is, the i * -th partial image 27 of the n * -th input image 1 is placed in the storage device 21, and the least usable partial image on the storage device 21 is saved in the secondary storage device 48 instead. . As a measure of the usability, a moving distance of the viewpoint 5 of the composite image 4 necessary for using each partial image is adopted. As shown in FIG. 20, when the number of input images 1 is sufficiently large, for a certain partial image stored in the storage device 21, this partial image is used for the composite image 4 because the line-of-sight vector 51 is This is a time when the line of sight vector 34 of the partial image 32 of No. 4 is on the same straight line in the same direction. When the line of sight vector 2 is not on the same straight line, the line of sight vector 2
9 will be used. Therefore, the distance between the straight line 52 on which the line-of-sight vector 51 is placed and the viewpoint 5 of the composite image 4 is defined as the moving distance. The smaller the moving distance, the higher the possibility that the partial image will be used. With respect to the partial images stored in the storage device 21, the partial image having the longest moving distance, that is, the least likely to be used partial image is saved in the secondary storage device 48.

In step ST82, a variable a indicating the availability of the partial image 27 is initialized to -1. Step S
In T83, it is determined whether or not there is an unprocessed partial image 27 in the storage device 21. If there is, the process proceeds to step ST84. If not, the process proceeds to step ST89. Step ST8
In step 4, for one unprocessed partial image of the input image 1 stored in the storage device 21, the number of the input image is stored in n1 and the number of the partial image is stored in i1. In step ST85, variables θ and ρ are extracted from n1 and i1 and are set as θ1 and ρ1. In step ST86, for the partial image represented by θ1 and ρ1, the viewpoint 5 of the composite image 4 necessary for using the partial image
Is calculated as follows (see FIG. 20). Assuming that the x coordinate of the viewpoint 5 is x and the z coordinate is z,

[0065]

(Equation 10)

In step ST87, it is determined whether or not d is larger than a. If d is larger, the process proceeds to step ST88, and if not, the process proceeds to step ST83. In step ST88, a is updated, and the number n1 of the processed input image 1 and the number i1 of the partial image 27 are stored. In step ST89, the partial image 27 specified by Nn and Ni, that is, the partial image 27 having the largest value represented by Expression 10 is moved from the storage device 21 to the secondary storage device 48. In step ST90, the i * -th partial image 2 of the n * -th input image 1 used to generate the composite image 4
7 is moved from the secondary storage device 48 to the storage device 21. In step ST91, n * used to generate the composite image 4
The i * -th partial image 27 of the first input image 1 is read from the storage device 21. In step ST92, the i * -th partial image 27 of the n * -th input image 1 is transferred to the partial image 32 of the composite image 4 by the above equations (6) to (10).

By the above processing, even when a large number of input images 1 are used, the number of times of reference to the secondary storage device 48 can be reduced, and an image synthesizing apparatus that can generate the synthesized image 4 at high speed can be obtained.

Embodiment 5 A block diagram showing a fifth embodiment of the present invention is the same as FIG. FIGS. 21 and 22 are flowcharts showing the operation of the fifth embodiment of the present invention. FIGS. 23 and 24 are explanatory diagrams showing the operation of the fifth embodiment of the present invention. 55 is a cursor displayed on the display device 26, 56 is a region where the focusing distance is set to D1 on the composite image 4,
Reference numeral 57 denotes an area in which the focus distance on the composite image 4 is set to D2.

In the image synthesizing apparatus configured as described above, when the objects 15 and 16 having different distances from the viewpoint 5 are present in the synthesized image 4, the images that accurately represent both are synthesized. Like the first embodiment, first,
The image 1 and accompanying information are input and stored in the storage device 21. This operation is the same as in the flowchart of FIG.

Next, the operation of synthesizing an image having a new viewpoint and a new optical axis direction will be described with reference to the flowchart of FIG. Each step performs the same operation as the flowchart of the first embodiment shown in FIG. 3, but the execution position of step ST21 is different from that of the first embodiment. Step ST21 in the fifth embodiment is executed each time each partial image 32 of the composite image 4 is generated. In step ST21, the focus adjustment device 23 inputs a distance D for focusing from the position of the viewpoint 5 of the composite image 4. An example in which the focus adjustment device 23 is realized by a cursor operation using a mouse or the like and a keyboard will be described with reference to the flowchart of FIG. 22 and FIG. In this example, the partial image 32 whose D value is to be changed is indicated by the cursor 55, and the increase or decrease of the D value is input from the keyboard. In step ST100,
The value of D used to generate the io-th partial image of the previous composite image 4 is read from the array Z. In the case of the first generation of the composite image 4, a predetermined initial value of D is used.
In step ST101, the value xc of the x coordinate of the display position of the cursor 55 on the display device 26 is obtained. In step ST102, the x-coordinate value xo of the position on the screen of the composite image 4 displayed on the display device 26 is obtained.
In step ST103, the cursor 55 is moved to any of the partial images 32 on the composite image 4 according to the position of the composite image 4 on the screen.
Calculate what is above. this is,

[0071]

[Equation 11]

Then, an i-th partial image 32 is obtained. In step ST104, it is determined whether or not i indicating the partial image pointed by the cursor 55 matches io indicating the transfer processing position of the composite image 4, and if so, step ST104 is performed.
The process proceeds to 105, otherwise proceeds to step ST107. In step ST105, it is checked whether or not there is an instruction to change the value of the distance D. For example, the key input of the keyboard is checked. If there is a specific input, it is determined that there is an instruction, and the process proceeds to step ST106.
Proceed to T107. In step ST106, the value of the distance D is updated. This can be realized by, for example, increasing or decreasing the value of D by key input on a keyboard. In step ST107, the value of D is stored in the array Z. For the transfer in step ST25, the D
Is used.

By executing as described above, it becomes possible to change the value of D for each partial image 32 of the composite image 4, and as shown in FIG. A composite image 4 that accurately represents an object can be obtained. The area 56 of the composite image 4 has the distance from the viewpoint 5 set to D1, and the area 57 has the distance from the viewpoint 5 set to D2. As a result, both the object 15 at the distance D1 and the object 16 at the distance D2 are accurately represented.

In the above embodiment, D is used as it is in each partial image of the composite image, that is, the composite image is focused on a cylindrical surface having a radius D centered on the viewpoint. In the generation of the partial image 32 which is the i-th vertical component of the composite image 4 instead of

[0075]

(Equation 12)

A composite image focused on a plane at a distance D from the viewpoint may be obtained by using D ′ represented by The focal plane can be freely adjusted by appropriately setting the above equation (12).

Further, in the above embodiment, the selection of the input partial image with respect to the composite image is performed by using the line-of-sight vector and the variables θ, ρ, ε and Equation 5, but this is performed by another method.
For example, you may comprise so that it may perform by the method shown in the said prior art document.

Further, in the above embodiment, all the variables ε have been obtained in advance for the input partial image. However, if the variable is not used for selecting the input partial image, the enlargement / reduction may be performed as in step ST25. You may comprise so that it may calculate at the time of performing.

In the above embodiment, the variable ε is used for the input partial image. However, when the variable is not used for selecting the input partial image, if the variable ε is not used for selecting the input partial image, step ST 25
For example, the enlargement / reduction performed by, for example, calculating the distance between the viewpoint positions of the input image 1 and the synthesized image 4 and using this value as Δε shown in Expression 4 or the like may be adopted.

In the third embodiment, one of the boundary areas is mixed with the other input image. However, the other input image is mixed on both sides of the boundary area. May be configured.

In the third embodiment, the mixing ratio of the input image at the boundary portion is changed linearly. However, the mixing ratio may be changed in another manner.

In the fourth embodiment, the partial image 27 of the input image 1 is stored in the storage device 21 and the secondary storage device 48.
However, all of them may be stored in the secondary storage device 48, and the portion used for the synthesis of the synthesized image 4 may be sequentially copied to the storage device 21.

Also, in the fourth embodiment,
Is used as a scale, but the determination is made using the difference between the visual axis direction of the composite image and the value of the variable θ in consideration of the optical axis direction and the angle of view of the composite image 4. May be configured. Further, the distance between the viewpoint of the synthesized image and the viewpoint of the input image may be used.

In the fourth embodiment, the storage devices 21 and 2 are used in units of the partial image 27 of the input image 1.
Although storage in the next storage device 48 is performed, a configuration may be adopted in which a partial image group in which a plurality of adjacent partial images of the same input image are grouped is treated as a unit. In addition, it is also possible to adopt a configuration in which clustering is performed on the different input images using the variables θ, ρ, and ε, and the partial images 27 having the same cluster are collectively handled as a partial image group. At this time, the minimum value of the distance d shown in Expression 10 of each of the partial images in the partial image group is set as the moving distance of the viewpoint 5 with respect to the partial image group, and step ST in FIG.
At 89, the partial image group having the largest moving distance is saved to the secondary storage device 48. At step ST90, the partial image group including the i * -th partial image of the n * -th input image is moved to the storage device 21. It is configured so that With such a configuration, the number of occurrences of data transfer between the storage device 21 and the secondary storage device 48, that is, the number of occurrences of steps ST89 and The number of executions of step ST90 can be reduced.

In the fourth embodiment, the main storage device of the computer is used as the first storage device, and the disk device is used as the second storage device. However, the texture memory of the computer is used as the first storage device. The same effect can be obtained by using a graphics memory and a main storage device or another external storage device as the second storage means.

In the above embodiment, the number of input images is limited and the input processing is performed first. However, the input processing and the synthesis processing are performed in parallel, and the number of input images is reduced. Without limitation, a video camera or the like may be used to input data as needed.

In the above embodiment, the image input device 17, the viewpoint information input device 18, the input image processing device 19,
Eye gaze determining device 20, viewpoint indicating device 22, focus adjusting device 2
3. The image synthesizing apparatus is configured by using the synthesized image generating apparatus 24, the searching apparatus 25, the image smoothing apparatus 40, and the storage control apparatus 49. All or a part of these processes is executed by software of a computer. You may comprise.

In the above embodiment, the display device 26 is used to display the composite image 4. However, the composite image 4 may be stored in the storage device 21.

[0089]

Since the present invention is configured as described above, it has the following effects.

According to the first aspect of the present invention, even when the distance to the object is not known in advance or when there are a plurality of objects having different distances, a composite image that accurately represents the target object is generated. Will be able to

According to the second aspect of the present invention, it is possible to generate a composite image that accurately represents a target object by enlarging or reducing a partial image in a single direction.

According to the third aspect of the present invention, when the distance to the object is not known in advance, when there are a plurality of objects with different distances, or when the partial image of the input image is Even when the image becomes smaller than the image, a composite image that accurately represents the target object can be generated.

According to the fourth aspect of the present invention, it is possible to reduce the number of times of searching for a partial image and generate a composite image that accurately represents a target object with a small amount of trouble.

According to the fifth aspect of the invention, it is possible to easily and interactively adjust the distance value to the object to be accurately represented.

According to the sixth aspect of the present invention, it is possible to generate a composite image that simultaneously and accurately represents a plurality of objects at different distances.

According to the seventh aspect of the present invention, it is possible to easily obtain the value of the distance between the viewpoint of the input image and the viewpoint of the composite image necessary for accurately representing the target object.

According to the eighth aspect of the present invention, even when a composite image is generated by transferring from a plurality of input images, a composite image having no discontinuity in luminance value can be obtained.

According to the ninth aspect of the present invention, even when a synthesized image is generated by transferring from a large number of input images and an area to be transferred from the same input image is reduced, a synthesized image having no discontinuity in luminance value can be obtained. It can be easily obtained.

According to the tenth aspect, even when a large number of input images are used, a composite image can be generated at high speed.

According to the eleventh aspect, it is possible to reduce the number of times of movement of the partial image of the input image between the storage devices, and to generate a composite image at a high speed.

According to the twelfth aspect of the present invention, it is possible to reliably select a partial image of an input image which is unlikely to be used after moving the viewpoint of a composite image and save the partial image from a high-speed storage device.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a first embodiment of the present invention.

FIG. 2 is a flowchart showing the operation of the first embodiment of the present invention.

FIG. 3 is a flowchart showing the operation of the first embodiment of the present invention.

FIG. 4 is an explanatory diagram showing an operation of the first embodiment of the present invention.

FIG. 5 is an explanatory diagram showing an operation of the first embodiment of the present invention.

FIG. 6 is an explanatory diagram showing the operation of the first embodiment of the present invention.

FIG. 7 is an explanatory diagram showing the operation of the first embodiment of the present invention.

FIG. 8 is an explanatory diagram showing an operation of the second embodiment of the present invention.

FIG. 9 is a flowchart showing the operation of the second embodiment of the present invention.

FIG. 10 is a block diagram showing Embodiment 3 of the present invention.

FIG. 11 is a flowchart showing the operation of the third embodiment of the present invention.

FIG. 12 is a flowchart showing a part of the operation of the third embodiment of the present invention.

FIG. 13 is an explanatory diagram showing the operation of the third embodiment of the present invention.

FIG. 14 is an explanatory diagram showing the operation of the third embodiment of the present invention.

FIG. 15 is an explanatory diagram showing the operation of the third embodiment of the present invention.

FIG. 16 is a block diagram showing a fourth embodiment of the present invention.

FIG. 17 is a flowchart showing the operation of the fourth embodiment of the present invention.

FIG. 18 is a flowchart showing the operation of the fourth embodiment of the present invention.

FIG. 19 is a flowchart showing a part of the operation of the fourth embodiment of the present invention.

FIG. 20 is an explanatory diagram showing the operation of the fourth embodiment of the present invention.

FIG. 21 is a flowchart showing the operation of the fifth embodiment of the present invention.

FIG. 22 is a flowchart showing a part of the operation of the fifth embodiment of the present invention.

FIG. 23 is an explanatory diagram showing the operation of the fifth embodiment of the present invention.

FIG. 24 is an explanatory diagram showing an example of a composite image according to the fifth embodiment of the present invention.

FIG. 25 is an explanatory diagram showing the operation of a conventional image synthesizing device.

FIG. 26 is an explanatory diagram showing the operation of a conventional image synthesizing device.

FIG. 27 is an explanatory diagram showing an example of a synthesized image by a conventional image synthesis device.

[Explanation of symbols]

1 input image, 1a input image having intersection 10 as a viewpoint,
2 viewpoint from which input image 1 was captured, 3 optical axes from which input image 1 was captured, 4 synthesized images to be newly synthesized, 5 synthesized images 4
6, the optical axis of the synthesized image 4, the projection plane of the synthesized image 4, the area corresponding to the i-th vertical component on the projection plane 7 of the synthesized image 4, the line of sight connecting the viewpoint 5 and the pixel 8. Straight line, intersection of line of sight line 9 and x-axis, 12, 12a
Projection planes of the input images 1 and 1a, 13 an i-th vertical component of the input image 1a, 14 an i-th vertical component of the composite image 4, 15 an object to be displayed in the composite image 4, 16 an object 15 Are second objects at different z-coordinate positions, 1
7 image input device, 18 image information input device, 19 input image processing device, 20 gaze determination device, 21 storage device,
Reference Signs List 22 viewpoint indicating device, 23 focus adjusting device, 24 synthetic image generating device, 25 searching device, 26 display device, 27 input partial image obtained by decomposing input image 1, 28
A portion 29 in the projection plane 12 corresponding to the input partial image 27;
Eye vector of input partial image 27, 30 origin, 31
Leg of a perpendicular drawn down from the origin 30 to a straight line on which the line-of-sight vector is placed, 32 a partial image obtained by decomposing the image 4 to be synthesized
A portion corresponding to the partial image 32 in the projection plane 7, a line-of-sight vector of the partial image 32, a cylindrical surface for focusing, 36 a partial image obtained by decomposing the image 4 to be synthesized by a width L,
37 a portion corresponding to the partial image 36 in the projection plane 7, 38
A partial image of the input image whose line-of-sight vector matches the partial image 36; a region 39 near the partial image 38 of the input image;
0 image smoothing device, 41 area generated by the same input image in composite image 4, 42 area 4 in composite image 4
43 adjacent to 1 and generated by another input image
Boundary between region 41 and region 42, region to be smoothed 44, smoothed synthesized image 45, synthesized image 4
An area adjacent to the middle area 42 and generated by another input image; 47 a boundary between the area 42 and the area 46; 48 secondary storage device; 49 storage control device;
51, the line of sight of the partial image 27 of the input image 1 stored in the storage device 21, 52 the line of sight on which the line of sight vector 51 is placed, 53 the viewpoint of the input image 1 stored in the storage device 21, 54 the synthesized image 4, an area generated by a certain input image 1; 55, a cursor displayed on the display device 26; 56, an area on the composite image 4 where the distance from the viewpoint 5 is set to D1; 57, a viewpoint 5 on the composite image 4 Area where the distance from is set to D2.

────────────────────────────────────────────────── ─── Continuation of the front page (56) References JP-A-5-88601 (JP, A) JP-A-6-34343 (JP, A) JP-A-7-200876 (JP, A) JP-A-7-2008 225855 (JP, A) JP-A-6-52291 (JP, A) Katsuyuki Kamei et al., "Generation of virtual space based on images", IEICE Technical Report, IEICE, 1996 May 28, Vol. 96, No. 82, p. 25-34 Katsuyuki Kamei and 3 others, "Generation of human motion animation by editing real images and 3D model of hand", IPSJ Journal, Information Processing Society of Japan, February 15, 1995 36, No. 2, p. 374- 382 (58) Fields investigated (Int. Cl. ⁷ , DB name) G06T 17/40 H04N 5/262 G06T 1/00 CSDB (Japan Patent Office) JICST file (JOIS)

Claims (12)

(57) [Claims] 1. An image input means for inputting an image, an input means for inputting a viewpoint position, an optical axis direction, and an angle of view of an input image input by the image input means and a synthesized image to be synthesized, and the input image and the synthesis Image dividing means for dividing the image into a plurality of partial images, line-of-sight calculating means for calculating the direction and position of the line of sight corresponding to each of the partial images of the input image and the composite image, and Transfer means for searching one of the partial images of the input image according to the direction and position of the line of sight, enlarging or reducing the searched partial image of the input image to a partial image of the composite image, and By changing a variable that controls the magnification of enlargement or reduction when the transfer unit transfers the partial image of the input image to the partial image of the composite image, the correct position and size of the composite image can be obtained. And an adjusting means for adjusting the distance from the viewpoint of the synthesized image to the object to be drawn. 2. The image dividing means, when dividing the input image and the composite image into a plurality of partial images, respectively, divides the input image and the composite image vertically by a pixel having a width of 1, and the transfer means 2. The image synthesizing apparatus according to claim 1, wherein a partial image of a pixel having a width of 1 of the input image searched at that time is transferred while being enlarged or reduced in a vertical direction. 3. Image input means for inputting an image, input means for inputting a viewpoint position, an optical axis direction, and an angle of view of an input image input by the image input means and a synthesized image to be synthesized, and the input image and the synthesis. Image dividing means for dividing the image into a plurality of partial images, line-of-sight calculating means for calculating the direction and position of the line of sight corresponding to each of the partial images of the input image and the composite image, and On the other hand, one of the partial images of the input image is searched according to the direction and position of the line of sight, and the partial image of the searched input image, the magnification or reduction of the searched partial image, and the partial image of the composite image Transfer means for enlarging or reducing a partial image located in a neighborhood range determined by the size of the composite image to transfer the partial image, and a partial image of the input image by the transfer means Is changed from the viewpoint of the composite image up to the object to be drawn at the correct position and size by changing the variable that controls the enlargement or reduction magnification when transferring the composite image to the partial image of the composite image. An image synthesizing apparatus comprising: an adjusting unit that adjusts a distance. 4. The image dividing means, when dividing the input image and the composite image into a plurality of partial images, respectively, divides the input image into partial images vertically divided by a fixed number of pixels having a width of 1 or more, The composite image is divided into partial images vertically divided by a fixed number of pixels having a width of 2 or more, and the transfer unit converts the partial image of the input image searched for at the time of transfer and the partial image in the vicinity thereof in the vertical and horizontal directions. The image synthesizing apparatus according to claim 3, wherein the image is transferred after being enlarged or reduced at the same magnification in the direction. 5. The adjusting means receives an input of a distance from a viewpoint of the synthesized image to an object to be drawn in the synthesized image at an accurate position and size every time the synthesized image is synthesized. The distance between the viewpoint of the synthesized image and the viewpoint of the partial image of the searched input image is obtained, and the magnification of the enlargement or reduction at the time of transfer is determined using this and the value of the distance to the object input by the adjusting means. 5. The method according to claim 1, wherein the determination is performed for each transfer of the composite image to the partial image.
The image synthesizing device according to any one of the above. 6. The transfer means receives the input of the distance from the viewpoint of the composite image to an object to be rendered in the composite image at an accurate position and size for each partial image of the composite image as needed. Calculates the distance between the viewpoint of the composite image and the viewpoint of the retrieved partial image of the input image, and uses this and the distance value to the target set for each partial image of the composite image input by the adjusting means. 5. The image synthesizing apparatus according to claim 1, wherein a magnification of enlargement or reduction at the time of transfer is determined for each transfer of a partial image of the composite image. 7. The line-of-sight calculation means calculates a size 1 in the line-of-sight direction corresponding to each of the divided images of the input image and the composite image.
Of the inner product of the vector of
In the distance calculating means, by using the value of the inner product obtained by the visual line calculating means, by taking the difference between the value of the inner product of both the partial image of the composite image and the partial image of the input image,
7. The image synthesizing apparatus according to claim 5, wherein a distance between a viewpoint of the synthesized image and a viewpoint of a partial image of the searched input image is obtained. 8. An image input means for inputting an image, an input means for inputting a viewpoint position, an optical axis direction, and an angle of view of an input image input by the image input means and a synthesized image to be synthesized, and the input image and the synthesis. Image dividing means for dividing the image into a plurality of partial images, line-of-sight calculating means for calculating the direction and position of the line of sight corresponding to each of the partial images of the input image and the composite image, and Transfer means for searching one of the partial images of the input image according to the direction and position of the line of sight, enlarging or reducing the searched partial image of the input image to a partial image of the composite image, and The composite image is divided into regions by the input image used in the generation of each partial image of the composite image by the transfer means, and the boundary between the regions and the partial image of the composite image belonging to the range in the vicinity thereof For the image, a new partial image is selected from the input image used in the other area of the boundary, and the newly selected partial image is overwritten while mixing pixel values with the partial image of the already transferred composite image. An image synthesizing apparatus comprising: 9. The image dividing means, when dividing the input image and the composite image into a plurality of partial images, respectively, divides the input image and the composite image vertically by pixels having a width of 1 or more, and The means were used in the other region for the partial images in the vicinity of the boundary only in the region on one end side of the boundary with respect to each boundary in order from the one on the left or right end of the boundary of the composite image 9. The image synthesizing apparatus according to claim 8, wherein a partial image is selected from the input image, and the selected partial image is overwritten while mixing pixel values with the partial image of the composite image. 10. An image input unit for inputting an image, an input unit for inputting a viewpoint position, an optical axis direction, and an angle of view of an input image input by the image input unit and a synthesized image to be synthesized, and the input image and the synthesis. An image dividing unit that divides the image into a plurality of partial images, a line-of-sight calculating unit that calculates a direction and a position of a line of sight corresponding to each of the partial images of the input image and the composite image, and a line-of-sight calculating unit that divides the composite image into respective partial images. Transfer means for searching one of the partial images of the input image according to the direction and position of the line of sight, enlarging or reducing the searched partial image of the input image to a partial image of the composite image, and First storage means for storing a partial image of an input image and enabling high-speed reading and writing, and storing a partial image of another input image or partial images of all input images A large-capacity second storage unit, which is stored in the first storage unit when the transfer unit transfers a partial image of the input image that is not stored in the first storage unit when the viewpoint of the composite image changes. The partial image of the input image to be transferred instead of the partial image having the largest viewpoint movement amount of the composite image required for the use of the partial image among the partial images of the input image is stored in the first storage unit from the second storage unit. An image synthesizing apparatus, comprising: a storage control unit that stores data in a storage unit. 11. Image input means for inputting an image, input means for inputting a viewpoint position, an optical axis direction, and an angle of view of an input image input by the image input means and a synthesized image to be synthesized, and the input image and the synthesis. Image dividing means for dividing the image into a plurality of partial images, line-of-sight calculating means for calculating the direction and position of the line of sight corresponding to each of the partial images of the input image and the composite image, and Transfer means for searching one of the partial images of the input image according to the direction and position of the line of sight, enlarging or reducing the searched partial image of the input image to a partial image of the composite image, and First storage means for storing a partial image of an input image and enabling high-speed reading and writing, and storing a partial image of another input image or partial images of all input images A large-capacity second storage means, and a plurality of partial images of the input image are collectively stored as a partial image group in the first storage means or the second storage means. When a partial image of the input image that is not stored in the storage unit is transferred by the transfer unit, any one of the partial images in the partial image group of the input image stored in the first storage unit is used. The partial image group including the partial image of the input image to be transferred by the transfer unit is transferred from the second storage unit to the first storage unit instead of the partial image group in which the viewpoint movement amount of the composite image required for the An image synthesizing apparatus, comprising: a storage control unit for storing. 12. The storage control means obtains a distance between a line of sight of the partial image of the input image and a viewpoint of the composite image with respect to the partial image of the input image stored in the first storage means. The image synthesizing apparatus according to claim 10 or 11, wherein the amount of viewpoint movement of the synthesized image necessary for using the partial image is used.