JP7613040B2 - Image processing system, image processing method, and program - Google Patents

Description

The present invention relates to an image processing system, an image processing method, and a program.

Conventionally, there is a technology that overlays an image of a virtual object on an image of a real space captured by an actual imaging device (real imaging device), generates an image in which the virtual object can be viewed as if it actually exists in the real space, and displays the image on a display device.
For example, in the image processing system described in Patent Document 1, parameters relating to the color and brightness of a virtual light source in a virtual space are calculated from a real image captured in a real space.
Then, in the virtual space in which the calculated virtual light source parameters have been set, a three-dimensional model, which is a virtual object, is captured by a virtual imaging device (virtual camera), and the captured image is displayed superimposed on an image captured by a real imaging device.

In addition, in the image processing system described in Patent Document 2, captured images of the subject and background and imaging information at the time of capturing the images are acquired, and three-dimensional data indicating the shape of the subject and two-dimensional data indicating the background are acquired from a database based on the imaging information.
Then, based on each of the acquired three-dimensional data and two-dimensional data, a simulation image is generated in which a subject of the three-dimensional data is placed against the background of the two-dimensional data.

Thereafter, the density, color, blur, and camera exposure of the simulation image are corrected based on the differences between the captured images of the subject and background and the three-dimensional data and two-dimensional data of the simulation image.
In this way, the weighted average of the pixel values of each pixel in the simulation image is calculated according to the enlargement ratio of the captured image and the simulation image, thereby processing the simulation image in response to the user's operation.

Patent No. 5646263 Patent No. 4829269

However, in Patent Document 1, the definition of a virtual object image, which is an image of a virtual object, is constant regardless of the focus position of the imaging device.
For this reason, the virtual object image superimposed on the real space may be viewed unnaturally in the real space. For example, in an image captured of the real space, the out-of-focus range of the imaging device is blurred, but the virtual object image superimposed on the out-of-focus range is not blurred.
As a result, only the virtual object image is displayed with higher resolution than the image in real space, and the virtual object image is perceived as floating from its surroundings, and is displayed as an unnatural composite image.

In addition, in Patent Document 2, a simulation image is generated based on three-dimensional data of a subject and two-dimensional data of a background obtained from a database.
For this reason, this image processing system cannot be applied to subjects and backgrounds for which there is no three-dimensional data or two-dimensional data in the database.
In addition, since it is assumed that a simulation image of a subject in a real space is to be generated, it is not possible to generate a simulation image in which a virtual object of a subject that does not actually exist at an arbitrary position in the real space appears to exist in the real space.

The present invention has been made in consideration of such circumstances, and provides an image processing system, image processing method, and program that, when superimposing an image of a virtual object on a captured image of real space, generates a composite image that can be viewed as a more natural display by superimposing an image of a virtual object at an arbitrary position on a captured image of real space, regardless of the state of the captured scene or the image capture device.

The image processing system of the present invention includes a subject information estimation unit that estimates subject information including the position and orientation of the subject relative to the imaging device based on a subject image of the subject in a reference image captured by the imaging device, and a computer graphics (CG) unit that generates a CG image of the subject based on the subject information. The image processing apparatus is characterized in that it includes a subject CG image drawing unit that draws a subject CG image, which is a virtual object image (graphics) image; a virtual object placement position acquisition unit that acquires a placement position for arranging a virtual object image in a target captured image captured by the imaging device; and a resolution calculation unit that compares the subject image with the subject CG image to acquire a subject image resolution that is the resolution of the subject image, and also acquires the virtual object image resolution that corresponds to the second distance by using a correspondence relationship that determines the virtual object image resolution that is the resolution of the virtual object image to be synthesized into the target captured image from the subject image resolution, reference captured image capture parameters when the reference captured image is captured by the imaging device, target captured image capture parameters when the target captured image is captured by the imaging device, a first distance between the subject and the imaging device, and a second distance that is the virtual distance between the imaging device and the virtual object of the virtual object image when the virtual object image is synthesized into the target captured image.

The image processing system of the present invention is characterized by further comprising a composite image generation unit that adjusts the resolution of the virtual object image based on the virtual object image resolution, and generates a composite object image by arranging the resolution-adjusted virtual object image at a predetermined position in correspondence with the second distance in the target captured image.

The image processing system of the present invention is characterized in that the correspondence includes at least a first calculation of dividing a second difference, which is an absolute value of a difference between the second distance and a second focus position in the target captured image imaging parameters, by a first difference, which is an absolute value of a difference between the first distance and a first focus position in the reference captured image imaging parameters, to obtain a first numerical value, dividing the second distance by the first distance to obtain a second numerical value, and multiplying each of the first numerical value and the second numerical value by the subject image resolution, and is a relational equation for obtaining the virtual object image resolution from the subject image resolution.

The image processing system of the present invention is characterized in that the relational expression obtains a third numerical value by dividing a third difference, which is a difference between the first focus position and a first focal length in the reference captured image imaging parameters, by a fourth difference, which is a difference between the second focus position and a second focal length in the target captured image imaging parameters, obtains a fourth numerical value by dividing the square of the second focal length by the square of the first focal length, obtains a fifth numerical value by dividing a first F value in the reference captured image imaging parameters by a second F value in the target captured image imaging parameters, and obtains the virtual object image definition by multiplying the first numerical value by a second numerical value that multiplies each of the third numerical value, the fourth numerical value, and the fifth numerical value.

The image processing system of the present invention is characterized by further comprising a subject imaging information estimation unit that estimates the first focal length, the first F-number, and the first focus position in the reference captured image from the first distance and the subject image resolution of each of the subjects in the reference captured image in which multiple subjects are captured.

The image processing system of the present invention is characterized in that the second focus position is equal to the first focus position, and the second focal length and the second F-number in the target captured image imaging parameters are equal to the first focal length and the first F-number in the reference captured image imaging parameters.

The image processing system of the present invention is characterized in that the resolution calculation unit includes a storage unit in which the subject image resolution and the reference captured image imaging parameters are written and stored for each type of imaging device.

The image processing system of the present invention is characterized in that the resolution calculation unit compares the subject image with an adjusted subject CG image obtained by filtering the subject CG image with a weighted average filter, and calculates, as the subject image resolution, the first filter radius of the weighted average filter that equalizes the difference between the subject image and the adjusted subject CG image.

The image processing system of the present invention is characterized in that the resolution calculation unit calculates a second filter radius, which is the virtual object image resolution, from the first filter radius in accordance with the correspondence relationship, and adjusts the resolution of the virtual object image.

The image processing method of the present invention includes a subject information estimation process in which a subject information estimation unit estimates subject information including a position and orientation of the subject relative to an imaging device based on a subject image of the subject in a reference imaging image captured by the imaging device, a subject CG image drawing process in which a subject CG image drawing unit draws a subject CG image that is a CG image of the subject based on the subject information, a virtual object placement position acquisition process in which a virtual object placement position acquisition unit acquires a placement position for placing a virtual object image in a target imaging image captured by the imaging device, and a resolution calculation process in which a resolution calculation unit calculates the resolution of the subject by comparing the subject image with the subject CG image. The method includes a process of acquiring a subject image resolution, which is the resolution of an image, and using a correspondence relationship to obtain a virtual object image resolution, which is the resolution of a virtual object image to be composited into the target captured image, from the subject image resolution, reference captured image capture parameters when the reference captured image is captured by the imaging device, a first distance between the subject and the imaging device, and a second distance, which is the virtual distance between the imaging device and the virtual object of the virtual object image when the virtual object image is composited into the target captured image, to acquire the virtual object image resolution corresponding to the second distance.

The program of the present invention causes a computer to function as a subject information estimation means for estimating subject information including the position and orientation of the subject relative to the imaging device based on a subject image of the subject in a reference imaging image captured by the imaging device, a subject CG image drawing means for drawing a subject CG image, which is a CG image of the subject, based on the subject information, a virtual object placement position acquisition means for acquiring a placement position for placing a virtual object image in a target imaging image captured by the imaging device, and a resolution calculation means for comparing the subject image with the subject CG image to acquire a subject image resolution, which is the resolution of the subject image, and acquiring the virtual object image resolution corresponding to the second distance using a correspondence relationship between the subject image resolution, reference imaging image capture parameters when the reference imaging image was captured by the imaging device, a first distance between the subject and the imaging device, and a second distance, which is the virtual distance between the imaging device and the virtual object of the virtual object image when the virtual object image is composited into the target imaging image.

As described above, according to the present invention, when an image of a virtual object is superimposed on a captured image of real space, it is possible to provide an image processing system, image processing method, and program that generate a composite image that can be viewed as a more natural display, obtained by superimposing an image of a virtual object on an arbitrary position of a captured image of real space, regardless of the state of the captured scene or the image capture device.

1 is a block diagram showing an example of the configuration of an image processing system according to a first embodiment of the present invention. 10 is a diagram showing an example of the configuration of a marker table in a marker information storage unit 108. FIG. 4 is a diagram showing an example of the configuration of a subject image table in a subject image information storage unit 109. FIG. 2 shows an example of the configuration of a subject estimation information table in the subject image information storage unit 109. 2 shows an example of the configuration of a subject image definition information table in a subject image definition information storage unit 110. 2 shows an example of the configuration of a captured image information table in a captured image information storage unit 111. 13 is a flowchart showing an example of the operation of a process of synthesizing a blurred virtual object corresponding to a focus position at a position set by a user in a target captured image in the image processing system according to the present embodiment. 13 is a diagram showing an example of the configuration of a subject image table in a subject image information storage unit 109 when a plurality of types of imaging devices are present. FIG. 13 shows an example of the configuration of a subject estimation information table in the subject image information storage unit 109 when multiple types of imaging devices are present. 1 shows an example of the configuration of a subject image definition information table in the subject image definition information storage unit 110 when multiple types of imaging devices are present. 1 shows an example of the configuration of a captured image information table in the captured image information storage unit 111 when multiple types of imaging devices are present. 13 is a diagram showing an example of the configuration of a subject image definition table of a subject image definition information storage unit 110 according to the second embodiment. FIG. 1 shows an example of the configuration of a subject image definition information table in the subject image definition information storage unit 110 when multiple types of imaging devices are present. FIG. 13 is a block diagram showing an example of the configuration of an image processing system according to a third embodiment of the present invention.

First Embodiment
An example of the configuration of the image processing system in FIG. 1 will be described below with reference to the drawings.
Fig. 1 is a block diagram showing an example of the configuration of an image processing system according to a first embodiment of the present invention. In Fig. 1, the image processing system 100 includes a data input/output unit 101, a subject information estimation unit 102, a subject CG image generation unit 103, a resolution calculation unit 104, a virtual object arrangement position acquisition unit 105, a composite image generation unit 106, a display unit 107, a marker information storage unit 108, a subject image information storage unit 109, a subject image resolution information storage unit 110, a captured image information storage unit 111, and a virtual object storage unit 112.

The data input/output unit 101 inputs data such as captured image data from an external device such as an imaging device, and operation/control information from a user.
The data input/output unit 101 writes and stores, for example, data of a captured image (reference captured image) of a real space captured by a user using the imaging device 200 as the background of a subject (marker) in the subject image information storage unit 109. In addition, the data input/output unit 101 writes and stores data of a captured image of the background as a real image in which a virtual object is to be placed in the captured image information storage unit 111.

The subject is a standard marker whose shape and surface texture are known. The standard marker to be the subject is written and stored in the marker information storage unit 108 in advance.
Fig. 2 is a diagram showing an example of the configuration of a marker table in the marker information storage unit 108. In Fig. 2, the marker table has columns for marker identification information, a marker shape data index, and a texture data index for each record.

The marker identification information is information for individually identifying each standard marker. The marker shape data index is, for example, an address of a storage area in the marker information storage unit 108 where shape data (actual size two-dimensional data or three-dimensional data) indicating the shape of the standard marker is stored. The texture data index is, for example, an address of a storage area in the marker information storage unit 108 where image data indicating the texture of the standard marker is stored.
Here, the standard marker has a texture having an asymmetric pattern in the vertical and horizontal directions in order to correctly detect the orientation, such as the inclination, which will be described later.

Returning to FIG. 1, the data input/output unit 101 writes and stores in the subject image information storage unit 109 reference image capturing parameters, which are the capturing parameters when the reference image was captured, together with the data of the reference captured image.
Fig. 3 is a diagram showing an example of the configuration of a subject image table in the subject image information storage unit 109. In Fig. 3, the subject image table has columns for each record, which are reference captured image imaging parameters, such as focal length (f), focus position (S), F-number (N), and resolution, and a reference captured image data index.

The focal length indicates the focal length f set in the imaging device 200 when the reference captured image was captured. The focus position indicates the focus position S set in the imaging device 200 when the reference captured image was captured. The F-number indicates the F-number N set in the imaging device 200 when the reference captured image was captured. The resolution indicates the resolution set in the imaging device 200 when the reference captured image was captured. The reference captured image data index is, for example, an address indicating the area in the subject image information storage unit 109 where the data of the reference captured image is stored.

Returning to FIG. 1, when the marker identification information of the standard marker 300 used as the subject is input together with the reference captured image imaging parameters, the data input/output unit 101 outputs the marker identification information to the subject information estimation unit 102 .
The subject information estimation unit 102 reads out, from the marker information storage unit 108, image data of the shape 300C and texture 300T of the subject (standard marker) 300 corresponding to the marker identification information supplied from the data input/output unit 101.

Then, the subject information estimation unit 102 extracts a region of the subject image corresponding to the marker image in the reference captured image by using general pattern matching.
Furthermore, the subject information estimation unit 102 acquires, in the above-mentioned pattern matching process, distance information between the subject 300 and the imaging device 200 in the reference captured image, and posture information of the subject, as positioning information, using the marker image data (each of the actual size data of the shape and each of the actual size data of the texture) and the subject image extracted by this marker image.

The subject information estimation unit 102 writes and stores the marker identification information and the acquired subject arrangement information in the subject image information storage unit 109 .
Fig. 4 shows an example of the configuration of a subject estimation information table in the subject image information storage unit 109. In Fig. 4, the subject estimation information table has columns for each record, including marker identification information, and distance information, orientation information, and position information as arrangement information.
The marker identification information is identification information of the subject (standard marker) 300 used when acquiring the placement information. The distance information is information indicating the distance between the imaging device that captured the reference captured image and the subject 300. The orientation information is information indicating the orientation of the subject 300 in the reference captured image, such as the inclination from the standard orientation of the subject image in the reference captured image (for example, the angle between the reference line in the standard orientation stored in the marker information storage unit 108 and the reference line of the subject image in the reference captured image). The position information is information indicating coordinate values on a two-dimensional coordinate system in the reference captured image.

Returning to FIG. 1, the subject CG image generating unit 103 reads the resolution of the reference captured image from the subject image table in the subject image information storage unit 109, and also reads the distance information, orientation information, and position information from the subject estimation information table.
Furthermore, the subject CG image generating unit 103 reads data on the shape 300C and texture 300T of the standard marker from the marker information storage unit 108.
Then, the subject CG image generation unit 103 uses each of the shape 300C and texture 300T of the standard marker to generate a CG image (subject CG image) of the standard marker corresponding to the distance information and posture information as a CG image of the same image size and resolution as the subject image in the reference captured image.

The resolution calculation unit 104 uses the pixel values of each pixel in the subject image extracted from the reference captured image and the generated subject CG image to calculate the filter radius of the weighted average filter as the resolution. In this embodiment, for example, a Gaussian filter, which is a type of weighted average filter, is used. In this embodiment, the filter radius is an example of a so-called filter size, and is the number of pixels that indicates the pixel range used in the filter process, centered on the target pixel that is the subject of the filter process.

Here, the definition calculation unit 104 performs weighted average processing by changing the filter radius of each pixel of the subject CG image, and compares the pixel values of each pixel of the subject CG image resulting from the processing with the subject image.
The definition calculation unit 104 then determines the filter radius R that minimizes the sum of the differences (e.g., squared errors) between the pixel values of each pixel in the subject CG image and the subject image as the subject image definition, which is the definition of the subject image.

Next, the definition calculation unit 104 reads out the focal length f, the focus position S, and the F-number N from the subject image information storage unit 109 .
Furthermore, the definition calculation unit 104 reads out the distance D (first distance) from the subject image information storage unit 109 as distance information.
Then, the definition calculation unit 104 calculates the subject blur B by substituting the focal length f (first focal length), focus position S (first focus position), F-number N (first F-number), and distance D when the reference image is captured by the imaging device 200 into the following equation (1).
B=|D−S|*f ² /((D*N)*(Sf)…(1)
In equation (1), |D-S| is the absolute value of the difference (first difference) obtained by subtracting the focal position S from the distance D, and (S-f) is the difference (third difference) obtained by subtracting the focal length f from the focal position S.

Then, the definition calculation unit 104 divides the calculated filter radius R by the amount of blur B as the radius of the circle of confusion using the following equation (2) to calculate the radius of the circle of confusion, and calculates a relationship coefficient k, which is the ratio between the filter radius R and the amount of blur B of the subject.
k=R/B=R*((D*N)*(S-f)/(|D-S|*f ² )...(2)
The definition calculation unit 104 writes and stores the calculated subject blur amount B, filter radius R, and relation coefficient in the subject image definition information storage unit 110 .

Fig. 5 shows an example of the configuration of a subject image definition information table in the subject image definition information storage unit 110. In Fig. 5, the subject image definition information table has columns for each record, for the subject blur amount B, the subject image definition (filter radius R), and the relationship coefficient k.
The subject blur (B) is a numerical value calculated by substituting the focal length f, the focus position S, the F-number N, and the distance D when the reference captured image is captured by the imaging device 200 into the above-mentioned formula (1). The subject image resolution is the filter radius R estimated from the subject image in the reference captured image. The relation coefficient (k) is a numerical value calculated by dividing the subject image resolution (filter radius R) by the subject blur B in the above-mentioned formula (2).

Returning to FIG. 1 , the virtual object arrangement position acquisition unit 105 acquires object arrangement information, supplied via the data input/output unit 101, for arranging a virtual object image in a target captured image input by a user.
Here, the object placement information is, for example, position information, orientation information, and distance information in which the user places a virtual object image in a target captured image (a captured image including a reference captured image) selected by the user and displayed on the display unit 107. This distance information is set as a virtual distance (virtual distance) from the imaging device 200 that captured the target captured image to the virtual object to be placed. For example, it is the distance from a virtual imaging device placed in a virtual three-dimensional space to the position of the virtual object, and the virtual imaging device corresponds to the imaging device 200 that captured the target captured image in the real space as a position in the three-dimensional space.

The object placement information may be inputted numerically from a keyboard or the like, or the position may be selected by moving a cursor in the target captured image displayed on the display unit 107 .
For example, when displayed on the display unit 107, a three-dimensional virtual object may be placed in a virtual three-dimensional space corresponding to the target captured image, and the placement information of the virtual object in the target captured image in the placed state may be acquired in real time as object placement information for the virtual object image.

In addition, the resolution calculation unit 104 reads the target captured image imaging parameters when the target captured image was captured by the imaging device 200 from the captured image information storage unit 111 in correspondence with the target captured image to be superimposed and synthesized with the virtual object image.
Fig. 6 shows an example of the configuration of a captured image information table in the captured image information storage unit 111. In Fig. 6, the captured image information table has columns for each record, including target captured image identification information, target captured image imaging parameters such as lens focal length (f _a ), focus position (S _a ), F-number (N _a ) and resolution, and a target captured image index.

The target captured image identification information is identification information of the subject (standard marker) 300 used when acquiring the placement information. The focal length indicates the focal length f _a set in the imaging device 200 when the reference captured image was captured. The focus position indicates the focus position S _a set in the imaging device 200 when the reference captured image was captured. The F-number indicates the F-number N _a set in the imaging device 200 when the reference captured image was captured. The resolution indicates the resolution set in the imaging device 200 when the reference captured image was captured. The target captured image data index is, for example, an address indicating an area in the captured image information storage unit 111 where the data of the target captured image is stored.

1 , when the object arrangement information is supplied, the definition calculation unit 104 reads out each of the target captured image imaging parameters from the captured image information table in the captured image information storage unit 111. In addition, the definition calculation unit 104 reads out the relationship coefficient k from the subject image definition information storage unit 110.
Then, the definition calculation unit 104 calculates the virtual object image definition (filter radius r) by substituting the focal length f _a , the focus position S _a and the F-number N _a , the relation coefficient k and the distance d into the following formula (3): where the distance d is the position information in the object arrangement information, i.e., the distance between the virtual imaging device (virtual camera) corresponding to the imaging device 200 and the virtual object.
r=k*f _a ² |d-S _a |/(d*N _a *(S _a -f _a ))...(3)
In equation (3) (correspondence), |d-S _a | is the absolute value (second difference) of the difference obtained by subtracting the focal position S _a (second focal position) from the distance d (second distance), and (S _a - _{f a} ) is the absolute value (fourth difference) of the difference obtained by subtracting the focal length f _a (second focal length) from the focal position S _a .

The above formula (3) at least includes formula (5) (first calculation, described later) which obtains a numerical value (first numerical value) by dividing the absolute value (second difference) of the difference between the distance d (second distance) and the focal position S _a (second focal position) by the absolute value (first difference) of the difference between the distance D (first distance) and the focal position S (first focal position), obtains a numerical value (second numerical value) by dividing the distance d (second distance) by the distance D (first distance), and multiplies each of the first numerical value and the second numerical value by the filter radius R (subject image resolution), and is a relational formula for obtaining the filter radius r (virtual object image resolution) from the filter radius R (subject image resolution).

Moreover, the above formula (3) obtains a numerical value (third numerical value) by dividing the difference (third difference) between the focus position S (first focus position) and the focal length _f (first focal length) by the difference (fourth difference) between the focus position S _a (second focus position) and the focal length f _a (second focal length), obtains a numerical value (fourth numerical value) by dividing the square of the focal length f (first focal length), obtains a numerical value (fifth numerical value) by dividing the F-number N (first F-number) by the F-number N _a (second F-number), and multiplies the first calculation by a second calculation that multiplies each of the third numerical value, the fourth numerical value, and the fifth numerical value to obtain the filter radius r (virtual object image resolution).

When the synthetic image generation unit 106 receives the virtual object image definition from the definition calculation unit 104 , the synthetic image generation unit 106 reads out data of the target captured image from the captured image information storage unit 111 .
Then, the composite image generating unit 106 reads out image data of the virtual object image from the virtual object storage unit 112, and generates an object CG image in which the image size of the virtual object image is corrected to a size corresponding to the distance d from the imaging device.

Furthermore, the composite image generating unit 106 performs a process of changing the resolution of the object CG image to match the resolution of the target captured image.
The composite image generating unit 106 performs weighted average filtering on the object CG image with the changed resolution using a filter radius r of the virtual object image definition.
As a result, the object CG image is rendered into a blurred image state that is visually recognized corresponding to the distance d.
Then, the composite image generating unit 106 superimposes the object CG image on the target captured image in accordance with the position information and orientation information in the object placement information to generate a composite object image.

The display unit 107 is, for example, a liquid crystal display.
The display unit 107 displays the composite object image generated by the composite image generating unit 106 on the display screen.
Furthermore, as already mentioned, when acquiring the position information, posture information, and distance information in the object placement information, the display unit 107 acquires each of the position information, posture information, and distance information in the placement information by capturing an image of the virtual three-dimensional space from a predetermined position, and having the user move the virtual object in the virtual three-dimensional space with the target mouse and control the posture.

FIG. 7 is a flowchart showing an example of the operation of the image processing system according to this embodiment, which synthesizes a blurred virtual object corresponding to the focus position at a position set by the user in a target captured image.
In the following description, data of a plurality of target captured images is acquired in advance and written and stored in the captured image information storage unit 111. The target captured images are captured by the same imaging device 200 as the reference captured image.

Step S101:
The user places the subject 300 (standard marker) in a predetermined posture at a predetermined position and distance in the background reference space.

Step S102:
The user captures a reference captured image so that the subject 300 placed in the reference space is included.

Step S103:
A user inputs to the image processing system 100 data of a reference captured image and subject imaging parameters used when the reference captured image was captured.
The data input/output unit 101 writes the reference captured image into the subject image information storage unit 109, and writes and stores the subject imaging parameters used when the reference captured image was captured and the reference captured image data index in the subject captured image table.

Step S104:
The user inputs the marker identification information of the standard marker of the subject 300 to the image processing system 100 .
When the subject information estimation unit 102 receives the above-mentioned marker identification information via the data input/output unit 101, it reads image data of the shape and texture corresponding to the marker identification information from the marker information storage unit 108 and acquires the marker image.

The subject information estimation unit 102 reads out the resolution in the reference captured image imaging parameters from the subject captured image information table in the subject image information storage unit 109 .
Furthermore, the subject information estimation unit 102 reads out data of the reference captured image from the subject image information storage unit 109 .
Then, the subject information estimation unit 102 performs a matching process using the marker image on the reference captured image, and extracts a subject image from the reference captured image.
At this time, the subject information estimation unit 102 extracts each of the position information and orientation information of the subject image in the reference captured image, and the distance D between the imaging device 200 and the subject 300 as distance information.

The subject CG image generating unit 103 reads data of the shape 300C and the texture 300T from the marker information storage unit 108 corresponding to the marker identification information of the standard marker used on the subject 300, and acquires a marker image.
Then, the subject CG image generating unit 103 generates a subject CG image of the standard marker corresponding to the distance information and posture information from the acquired marker image as a CG image having the same image size and resolution as the subject image in the reference captured image.

Step S105:
When the definition calculation unit 104 is supplied with a subject CG image generated by the subject CG image generation unit 103, the definition calculation unit 104 performs weighted averaging processing on each pixel in the subject CG image while changing the filter radius, thereby adjusting the pixel value of the pixel.
The definition calculation unit 104 compares the pixel values of the pixels in the subject CG image whose pixel values have been adjusted with the pixel values of the pixels in the subject image.
The definition calculation unit 104 then determines the filter radius R that minimizes the sum of the differences in pixel values of the pixels between the subject CG image and the subject image as the subject image definition, which is the definition of the subject image.
The definition calculation unit 104 writes and stores the obtained subject image definition in a subject image definition information table in the subject image definition information storage unit 110 .

Step S106:
Next, the definition calculation unit 104 reads out the focal length f, the focus position S, and the F-number N when the subject image was captured from the subject captured image information table in the subject image information storage unit 109 .

Step S107:
The definition calculation unit 104 reads out the distance D from the subject estimation information table in the subject image information storage unit 109 as information about the distance from the imaging device to the subject.
The definition calculation unit 104 then substitutes the read focal length f, focus position S, F-number N, and distance D into the above equation (1) to obtain the amount of subject blur B.
Furthermore, the definition calculation unit 104 calculates the relationship coefficient k by dividing the calculated filter radius R by the amount of blur B using the above formula (2) for calculating the radius of the circle of confusion.
The definition calculation unit 104 writes and stores the calculated subject blur amount B and relation coefficient k in a subject image definition information table in the subject image definition information storage unit 110 .

Step S108:
The virtual object arrangement position acquisition unit 105 acquires, via the data input/output unit 101, object arrangement information for arranging a virtual object image in a target captured image.
At this time, the virtual object arrangement position acquisition unit 105 acquires, through an input operation by the user, target captured image identification information of the target captured image used to generate the composite image, and virtual object identification information indicating the virtual object.
Here, when the image is displayed on the display unit 107, an operation is performed to place a three-dimensional virtual object at any coordinate point in the virtual three-dimensional space corresponding to the target captured image that the user views from the display screen of the display unit 107.
Then, the virtual object arrangement position acquisition unit 105 acquires the coordinate points of the arrangement in real time as virtual object arrangement information (position information, orientation information, and distance information) in the target captured image.

Step S109:
The virtual object placement position acquisition unit 105 outputs the acquired virtual object placement information, the target captured image identification information of the target captured image into which the virtual object is to be combined, and the virtual object identification information to the definition calculation unit 104.
The definition calculation unit 104 reads out the target captured image shooting parameters corresponding to the target captured image identification information from the captured image information table in the captured image information storage unit 111 .

Furthermore, the definition calculation unit 104 reads out the relationship coefficient k from the subject image definition information table in the subject image definition information storage unit 110 .
Then, the resolution calculation unit 104 substitutes the distance information (distance d) in the virtual object placement information, the target captured image imaging parameters of focal length fa, focus position Sa, and F-number Na, and the relationship coefficient k into the above equation (3) to calculate the virtual object image resolution (filter radius r).
The definition calculation unit 104 outputs the calculated virtual object image definition, the target captured image identification information, and the virtual object identification information to the synthetic image generation unit 106.

Step S110:
When the composite image generating unit 106 receives target captured image identification information from the definition calculating unit 104 , the composite image generating unit 106 reads out data of the target captured image corresponding to the target captured image identification information from the captured image information storing unit 111 .
Furthermore, the composite image generating unit 106 reads out data of a virtual object image corresponding to the virtual object identification information from the virtual object storage unit 112 .
The composite image generating unit 106 then generates an object CG image by correcting the image size of the read virtual object to a size corresponding to the distance d from the imaging device.

Furthermore, the composite image generating unit 106 reads out the resolution of the target captured image used for composition from the captured image information table in the captured image information storage unit 111 based on the target captured image identification information.
The composite image generating unit 106 performs processing to change the resolution of the object CG image to that of the target captured image.
The composite image generating unit 106 performs weighted average filtering on the object CG image with the changed resolution using a filter radius r of the virtual object image definition.
Then, the composite image generating unit 106 generates a composite object image by combining the object CG image with the target captured image in accordance with the position information and orientation information in the object placement information.

Step S111:
The virtual object arrangement position acquisition unit 105 determines whether or not control information for fixing the arrangement position of the virtual object has been input by the user, that is, whether or not to determine the position of the virtual object in the target captured image.
At this time, if the user has determined the position of the virtual object, the virtual object placement position acquisition unit 105 advances the process to step S112.
On the other hand, if the user has not determined the position of the virtual object, the virtual object placement position acquisition unit 105 advances the process to step S108.

Step S112:
The composite image generating unit 106 writes the composite object image into the virtual object storage unit 112 for storage.

As described above, according to this embodiment, the relationship coefficient k is calculated based on the subject image resolution when a subject placed in real space is photographed, and the virtual object image resolution of the virtual object to be superimposed on the target captured image obtained by photographing the real space is determined from the relationship coefficient k. Therefore, when a virtual object is composited with the target captured image, the sense of incongruity caused by the difference in the degree of blur between the virtual object and the background of the target captured image can be reduced.

In addition, according to this embodiment, the relationship coefficient k corresponding to the reference captured image imaging parameters when the subject is captured is recorded as a table, so that the resolution of the virtual object can be adjusted in real time even if the reference captured image imaging parameters and the captured image imaging parameters of the target captured image are different.

In addition, in this embodiment, as another configuration example, if there is not one image capture device 200 but a plurality of different types, it is necessary to obtain the relationship coefficient k corresponding to each type of image capture device 200.
In this case, the subject image table in FIG. 3, the subject estimation information table in FIG. 4, and the subject image definition table in FIG. 5 are provided corresponding to the imaging device identification information, which is identification information of the imaging device.

FIG. 8 is a diagram showing an example of the configuration of a subject image table in the subject image information storage unit 109 when multiple types of imaging devices are present.
In Figure 8, when there are multiple types of imaging devices, the subject image table has columns for each record: imaging device identification information, reference imaging image imaging parameters such as focal length (f), focus position (S), F-number (N) and resolution, and a reference imaging image data index.

The imaging device identification information is information for individually identifying the imaging device 200 that captured the reference image. The focal length indicates the focal length f that was set in the imaging device 200 when the reference image was captured. The focus position indicates the focus position S that was set in the imaging device 200 when the reference image was captured. The F-number indicates the F-number N that was set in the imaging device 200 when the reference image was captured. The resolution indicates the resolution that was set in the imaging device 200 when the reference image was captured. The reference captured image data index is, for example, an address that indicates the area in the subject image information storage unit 109 where the data of the reference image is stored.

FIG. 9 shows an example of the configuration of a subject estimation information table in the subject image information storage unit 109 when multiple types of imaging devices are present.
In FIG. 9, the subject estimation information table in the case where there are multiple types of imaging devices has columns for imaging device identification information, marker identification information, and distance information, attitude information, and position information as arrangement information for each record.

The imaging device identification information is information that individually identifies the imaging device 200 that captured the reference image. The marker identification information is identification information of the subject (standard marker) 300 used when acquiring the placement information. The distance information is information that indicates the distance between the imaging device that captured the reference image and the subject 300. The orientation information is information that indicates the orientation, such as the inclination from the standard orientation of the subject image of the subject 300 in the reference image. The position information is information that indicates the coordinate values on the two-dimensional coordinate system in the reference image.

Figure 10 shows an example of the configuration of a subject image resolution information table in the subject image resolution information storage unit 110 when multiple types of imaging devices exist. In Figure 10, the subject image resolution information table when multiple types of imaging devices exist has columns for the subject blur amount B, subject image resolution (filter radius R), and relationship coefficient k for each record.

The imaging device identification information is information that individually identifies the imaging device 200 that captured the reference captured image. The subject blur (B) is a value calculated by substituting the focal length f, focus position S, F-number N, and distance D when the reference captured image was captured by the imaging device 200 into the above formula (1). The subject image resolution is the filter radius R estimated from the subject image in the reference captured image. The relationship coefficient (k) is a value obtained by dividing the subject image resolution (filter radius R) by the subject blur B in the above formula (2).

Furthermore, when there is not one but a plurality of different types of imaging devices 200, the target captured image is also captured by the plurality of imaging devices 200.
For this reason, the captured image information table of the captured image information storage unit 111 shown in FIG. 6 also needs to be generated in advance in correspondence with each type of imaging device 200.

Fig. 11 shows an example of the configuration of a captured image information table in the captured image information storage unit 111 when there are multiple types of imaging devices. In Fig. 11, the captured image information table is provided for each type of imaging device with imaging device identification information added. Each captured image information table has columns for target captured image identification information, target captured image imaging parameters such as lens focal length (f _a ), focus position (S _a ), F-number (N _a ) and resolution, and target captured image index for each record.

When there are multiple types of imaging devices 200, the user uses each of the imaging devices 200 to capture a reference captured image in which a subject (standard marker 300) is placed against the background of the real space.
Then, the data input/output unit 101 writes and stores the imaging device identification information, reference imaging image, and subject imaging parameters of different types of imaging devices 200 supplied from an external device in the subject image table of the subject image information storage unit 109 shown in Figure 8, corresponding to the imaging device identification information.

Furthermore, the subject information estimation unit 102 performs the matching process between the standard markers 300 and the subject images in the reference captured images, as described above, for each of the different types of imaging devices 200 .
Then, the subject information estimation unit 102 writes and stores the marker identification information of the standard marker 300 used on the subject and the placement information estimated from this marker information in the subject estimation information table of the subject image information storage unit 109, corresponding to each of the different types of imaging devices 200.

The subject CG image generating unit 103 generates a subject CG image of the marker image used when extracting the subject image from each of the reference captured images captured by the different types of imaging devices.
The definition calculation unit 104 then compares the subject images in the reference captured images of the different types of imaging devices 200 with the subject captured image generated from the reference captured images, and calculates the subject image definition (filter radius R).

Moreover, the definition calculation unit 104 calculates the subject blur amount B from the reference captured image capture parameters according to equation (1).
Furthermore, the definition calculation unit 104 calculates a relationship coefficient k from the subject blur amount B and the filter radius R using equation (2).
Then, the definition calculation unit 104 writes and stores the calculated subject blur B, subject image definition, and relationship coefficient k in a subject image definition information table in the subject image definition information storage unit 110, corresponding to each piece of imaging device identification information.

When arranging a virtual object, the virtual object arrangement position acquisition unit 105 selects a captured image from the captured image information table of the captured image information storage unit 111, and acquires imaging device identification information corresponding to the selected captured image from the user.
Then, the virtual object arrangement position acquisition unit 105 outputs the acquired imaging device identification information and the target captured image identification information of the selected captured image to the definition calculation unit 104.

The definition calculation unit 104 refers to the captured image information table in the captured image information storage unit 111 that corresponds to the imaging device identification information supplied from the virtual object arrangement position acquisition unit 105 .
Then, the definition calculation unit 104 reads out the target captured image capturing parameters of the record of the target captured image identification information from the captured image information table.
Furthermore, the definition calculation unit 104 refers to the subject image definition information table in the subject image definition information storage unit 110 and reads out the relationship coefficient k corresponding to the imaging device identification information.
As a result, the definition calculation unit 104 calculates the virtual object image definition (filter radius r) from equation (3) using the distance d from the imaging device to the virtual object and the relation coefficient k.
The definition calculation unit 104 outputs the calculated virtual object image definition, the imaging device identification information, and the target captured image identification information to the composite image generation unit 106 .

The composite image generating unit 106 refers to the captured image information table corresponding to the imaging device identification information in the captured image information storage unit 111, and reads out the resolution corresponding to the target captured image identification information.
Then, the composite image generating unit 106 reads image data of the virtual object from the virtual object storage unit 112 in association with the virtual object identification information of the virtual object selected by the user.

The composite image generating unit 106 changes the size of the image data to correspond to the distance d, and adjusts the resolution to be the same as that of the target captured image, thereby generating a virtual object image.
Furthermore, the composite image generating unit 106 performs filtering processing on the virtual object image using the virtual object image definition supplied from the definition calculating unit 104 .
The composite image generating unit 106 then arranges and composites the filtered virtual object image at a specified position in the target captured image in accordance with the set orientation, thereby generating a composite object image.

Second Embodiment
An example of the configuration of an image processing system according to a first embodiment of the present invention will be described with reference to the drawings. The image processing system according to the second embodiment has the same configuration as that of the first embodiment shown in FIG.
Hereinafter, the image processing system according to the second embodiment will be described only in terms of the configuration and operation that differ from the first embodiment.

The definition calculation unit 104 calculates a relationship coefficient q from the estimated subject image definition (filter radius R), the focal position S, and the estimated distance D according to the following equation (4).
q=R*D/(|D-S|)...(4)
In other words, the relationship coefficient q is calculated as the ratio of the product of the filter radius R and the distance D, obtained by multiplying the filter radius R by the distance D and dividing the product by the absolute value obtained by subtracting the focal position S from the distance D.

The definition calculation unit 104 then writes and stores the obtained relationship coefficient q, subject image definition, and focus position S in a subject image definition table in the subject image definition information storage unit 110 .
Fig. 12 is a diagram showing an example of the configuration of a subject image resolution table in the subject image resolution information storage unit 110 in the second embodiment. In Fig. 12, the subject image resolution information table has columns for each record, including the focus position S, the focal length f, the F-number N, the subject image resolution (filter radius R), and the relationship coefficient q.

Also, as described later, in this embodiment, the relationship coefficient q calculated at a specified focus position S is limited to calculate the virtual object image resolution of a virtual object image in a target captured image having approximately the same (equal) focus position S.
For this reason, since the desired virtual object image definition is fixed at the focus position S, it is necessary to capture reference captured images corresponding to a plurality of different focus positions S and generate the relation coefficient q.

The focal position (S) indicates the focal position S set in the imaging device 200 when the reference captured image was captured. The subject image resolution is the filter radius R estimated from the subject image in the reference captured image. The relationship coefficient (q) is a numerical value calculated by the above-mentioned formula (4) using the subject image resolution (filter radius R), the distance D estimated by the subject information estimation unit 102, and the focal position S.

In addition, the resolution calculation unit 104 calculates the virtual object image resolution (filter radius r) from the following equation (5) using the relationship coefficient q, the focus position S of the imaging device 200 when the reference captured image was captured, and the distance d set by the user.
r=(|d−S|/d)*q
=(|d-S|/d)*(R*D/|D-S|)...(5)

Unlike the case of the relationship coefficient k, the above-mentioned relationship coefficient q requires that the focus position S in the reference captured image imaging parameters and the focus position S in the target captured image imaging parameters are the same. In addition, in the present embodiment, the focal length f and the F-number N in the target captured image parameters also need to be the same as the focal length f and the F-number N in the reference captured image parameters, respectively.
Therefore, when selecting a target captured image to be used to generate an object composite image from the captured image information storage unit 111, it is necessary to search for captured images that have the same focus position S, focal length f, and F-number N as those used when calculating the relationship coefficient q, and then select from among the captured images found.
In addition, the algorithm for generating an object CG image using the virtual object image resolution adjusted to the resolution of the target captured image and calculated using the relationship coefficient q, and then synthesizing the object CG image with the target captured image to generate a composite object image is the same as in the first embodiment.

Also, in the second embodiment, similarly to the first embodiment, a plurality of different types of image capturing devices 200 may be used. When a plurality of different types of image capturing devices are used, it is necessary to provide the subject image resolution information table of Fig. 12 for each captured image.
FIG. 13 shows an example of the configuration of a subject image definition information table in the subject image definition information storage unit 110 when multiple types of imaging devices are present.
13, the captured image information table is provided for each type of imaging device, with imaging device identification information added, and each captured image information table has columns for the focus position S, focal length f, F-number N, subject image resolution (filter radius R), and relationship coefficient q for each record.

As described above, according to this embodiment, the relationship coefficient q is calculated based on the subject image resolution when a subject placed in real space is photographed, and the virtual object image resolution of the virtual object to be superimposed on the target captured image obtained by photographing the real space is determined from the relationship coefficient q. Therefore, when a virtual object is composited with a target captured image, the sense of incongruity caused by the difference in the degree of blur between the target captured image and the background can be reduced.

In addition, according to this embodiment, the relationship coefficient k corresponding to the reference captured image imaging parameters when the subject is captured is recorded as a table, so that the resolution of the virtual object can be adjusted in real time even if the reference captured image imaging parameters and the captured image imaging parameters of the target captured image are different.

Third Embodiment
An example of the configuration of the image processing system in FIG. 14 will be described below with reference to the drawings.
Fig. 14 is a block diagram showing an example of the configuration of an image processing system according to the third embodiment of the present invention. In Fig. 14, the image processing system 100 includes a data input/output unit 101, a subject information estimation unit 102, a subject CG image generation unit 103, a resolution calculation unit 104, a virtual object arrangement position acquisition unit 105, a composite image generation unit 106, a display unit 107, a marker information storage unit 108, a subject image information storage unit 109, a subject image resolution information storage unit 110, a captured image information storage unit 111, a virtual object storage unit 112, and a subject captured image information estimation unit 113. Hereinafter, only the configuration and operation of the image processing system 100A according to the third embodiment that are different from those of the first embodiment will be described.

The data input/output unit 101 writes and stores data of a captured image (reference captured image) of a real space with a subject (marker) as the background, captured by a user using the imaging device 200, in the subject image information storage unit 109. In this embodiment, multiple standard markers (e.g., standard markers 301, 302, 303, and 304) are used, and therefore the multiple standard markers are positioned at different distances in one reference captured image and captured.

Here, each of the multiple standard markers may be either the same or different standard markers. Also, a configuration may be adopted in which multiple standard markers are arranged at different distances with the same focus position S, focal length f, and F-number S, reference captured images are captured for each different distance, and simultaneous linear equations (described later) are generated using each of the multiple reference captured images.
The subject information estimation unit 102 reads out from the marker information storage unit 108 image data of the shapes 301C, 302C, 303C, and 303C of the subjects (standard markers) 301, 302, 303, and 304, respectively, and the textures 301, 302T, 303T, and 304T, which correspond to the marker identification information supplied from the data input/output unit 101.

Then, the subject information estimation unit 102 extracts areas of the subject image corresponding to the marker images of the standard markers 301, 302, 303, and 304 in the reference captured image by using general pattern matching.
Furthermore, in the above-mentioned pattern matching process, the subject information estimation unit 102 acquires, as positioning information, distance information d1, d2, d3 between each of the standard markers 301, 302, 303, and 304 in the reference captured image and the imaging device 200, as well as posture information of the subject, using the marker image data (each of the actual size data of the shape and each of the actual size data of the texture) and the subject image extracted from this marker image.
The subject information estimation unit 102 writes and stores each of the distance information d1, d2, and d3 and the subject's posture information in a subject estimation information table in the subject image information storage unit 109.

Next, the subject CG image generating unit 103 reads out the resolution of the reference captured image from the subject image table in the subject image information storage unit 109, and also reads out the distance information, orientation information, and position information from the subject estimation information table.
Furthermore, the subject CG image generating unit 103 reads data on the standard marker shapes 301C, 302C, 303C, and 304C and textures 301T, 302T, 303T, and 304T corresponding to the marker identification information of each of the standard markers 301, 302, 303, and 304 from the marker information storage unit 108.
Then, the subject CG image generation unit 103 uses the standard marker shapes 301C, 302C, 303C, and 303C and the textures 301T, 302T, 303T, and 304T to generate CG images (subject CG images) of the standard markers 301, 302, 303, and 304 corresponding to the distance information and posture information as CG images having the same image size and resolution as the subject image in the reference captured image.

The resolution calculation unit 104 calculates the filter radius of a weighted average filter as the resolution using the pixel values of each pixel in the subject image extracted from the reference captured image and each of the generated subject CG images of the standard markers 301, 302, 303, and 304.
Then, the resolution calculation unit 104 determines the filter radius R that minimizes the sum of the differences (e.g., squared errors) between the pixel values of each pixel in the subject CG image and the subject image, for each of the standard markers 301, 302, 303, and 304, and determines this as the subject image resolution, which is the resolution of the subject image.

Then, the subject imaging information estimation unit 113 uses the following equation (6) to generate simultaneous linear equations that estimate the focal length f, focus position S, F-number N, and relationship coefficient k when the imaging device 200 captured the reference captured image.
R _i =k*f ² |d _i -S|/(d _i *N*(S-f))...(6)
In the above formula (6), R _i is the subject definition of the standard marker i, and d _i is the distance between the standard marker i and the image capturing device 200. In the case of this embodiment, standard markers 301, 302, 303, and 304 are used.

Therefore, the subject imaging information estimation unit 113 generates the following simultaneous linear equations using the subject definitions R ₃₀₁ , R ₃₀₂ , R ₃₀₃ and R ₃₀₄ and the distances d ₃₀₁ , d ₃₀₂ , d ₃₀₃ and d ₃₀₄ , where i is 301≦i≦304.
R ₃₀₁ =k*f ² |d ₃₀₁ -S|/(d ₃₀₁ *N*(S-f))
R ₃₀₂ =k*f ² |d ₃₀₂ -S|/(d ₃₀₂ *N*(S-f))
R ₃₀₃ =k*f ² |d ₃₀₃ -S|/(d ₃₀₃ *N*(S-f))
R ₃₀₄ =k*f ² |d ₃₀₄ -S|/(d ₃₀₄ *N*(S-f))

Moreover, the subject imaging information estimation unit 113 estimates the focal length f, the focus position S, the F-number N, and the relationship coefficient k by performing a calculation process to solve the simultaneous linear equations.
Then, the composite image generation unit 106 uses the focal length f, the focus position S, the F-number N, and the relationship coefficient k to read out image data of the virtual object image from the virtual object storage unit 112 according to equation (3), as in the first embodiment, and generates an object CG image in which the image size of the virtual object image is corrected to a size corresponding to the distance d from the imaging device.

As described above, according to this embodiment, as in the first embodiment, the relationship coefficient k is calculated based on the subject image resolution when a subject placed in real space is photographed, and the virtual object image resolution of the virtual object to be superimposed on the target captured image obtained by photographing the real space is determined from the relationship coefficient k. Therefore, when a virtual object is composited with a target captured image, it is possible to reduce the sense of incongruity caused by the difference in the degree of blur between the target captured image and the background.

In addition, according to this embodiment, the subject imaging information estimation unit 113 estimates the focal length f, the focus position S, the F-number N, and the relationship coefficient k from the marker images of multiple standard markers, so that the user does not need to check the focal length f, the focus position S, and the F-number N each time he or she captures a reference captured image with an imaging device, thereby reducing the effort required for the imaging process of the reference captured image.

The above describes an embodiment of the present invention with reference to the drawings, but the specific configuration is not limited to this form and includes designs that do not deviate from the gist of the invention.

Note that a program for implementing the functions of the image processing system 100 of FIG. 1 of the present invention may be recorded on a computer-readable recording medium, and the program recorded on the recording medium may be read into a computer system and executed to composite a virtual object with an image of real space after performing a filter process corresponding to the degree of blur in the captured image. Note that the term "computer system" here includes hardware such as an OS and peripheral devices.

"Computer system" also includes a WWW system equipped with a homepage provision environment (or display environment). "Computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems. "Computer-readable recording medium" also includes devices that hold a program for a certain period of time, such as volatile memory (RAM) inside a computer system that serves as a server or client when a program is transmitted via a network such as the Internet or a communication line such as a telephone line.

The above program may also be transmitted from a computer system in which the program is stored in a storage device or the like to another computer system via a transmission medium, or by transmission waves in the transmission medium. Here, the "transmission medium" that transmits the program refers to a medium that has the function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line. The above program may also be one that realizes part of the above-mentioned functions. Furthermore, it may be a so-called difference file (difference program) that can realize the above-mentioned functions in combination with a program already recorded in the computer system.

Reference Signs List 100 Image processing system 101 Data input/output unit 102 Subject information estimation unit 103 Subject CG image generation unit 104 Resolution calculation unit 105 Virtual object placement position acquisition unit 106 Composite image generation unit 107 Display unit 108 Marker information storage unit 109 Subject image information storage unit 110 Subject image resolution information storage unit 111 Captured image information storage unit 112 Virtual object storage unit 113 Subject captured image information estimation unit 113
200: Imaging device 300: Standard marker 300C: Shape 300T: Texture

Claims (11)

a subject information estimation unit that estimates subject information including a position and an orientation of the subject relative to an imaging device based on a subject image of the subject in a reference captured image captured by the imaging device;
a subject CG image drawing unit that draws a subject CG image, which is a CG (computer graphics) image of the subject, based on the subject information;
a virtual object position acquisition unit that acquires a position at which a virtual object image is to be arranged in a target captured image captured by the imaging device;
and a resolution calculation unit that compares the subject image with the subject CG image to acquire a subject image resolution, which is the resolution of the subject image, and also acquires the virtual object image resolution corresponding to the second distance using a correspondence relationship to obtain a virtual object image resolution, which is the resolution of a virtual object image to be synthesized with the target captured image, from the subject image resolution, reference captured image capture parameters when the reference captured image was captured by the imaging device, target captured image capture parameters when the target captured image was captured by the imaging device, a first distance between the subject and the imaging device, and a second distance, which is a virtual distance between the imaging device and the virtual object of the virtual object image when the virtual object image is synthesized with the target captured image. 2. The image processing system according to claim 1, further comprising a composite image generation unit that adjusts the resolution of the virtual object image based on the virtual object image resolution, and generates a composite object image by placing the resolution-adjusted virtual object image at a predetermined position corresponding to the second distance in the target captured image. The correspondence relationship is
The image processing system according to claim 1 or claim 2, further comprising at least a first calculation of: dividing a second difference, which is an absolute value of a difference between the second distance and a second focus position in the target captured image imaging parameters, by a first difference, which is an absolute value of a difference between the first distance and a first focus position in the reference captured image imaging parameters, to obtain a first numerical value; dividing the second distance by the first distance to obtain a second numerical value; and multiplying each of the first numerical value and the second numerical value by the subject image resolution, wherein the image processing system is a relational equation for obtaining the virtual object image resolution from the subject image resolution. The above-mentioned relational expression is
4. The image processing system according to claim 3, further comprising: a third difference, which is a difference between the first focus position and a first focal length in the reference captured image imaging parameters, divided by a fourth difference, which is a difference between the second focus position and a second focal length in the target captured image imaging parameters, to obtain a third numerical value; a square of the second focal length divided by the square of the first focal length to obtain a fourth numerical value; a first F-value in the reference captured image imaging parameters divided by a second F-value in the target captured image imaging parameters to obtain a fifth numerical value; and a second operation, which multiplies each of the third numerical value, the fourth numerical value, and the fifth numerical value, by the first operation to obtain the virtual object image resolution. 5. The image processing system according to claim 4, further comprising a subject imaging information estimation unit configured to estimate the first focal length, the first F-number, and the first focus position in the reference captured image from the first distance and the subject image resolution of each of the subjects in the reference captured image in which a plurality of subjects are captured. The image processing system of claim 3, characterized in that the second focus position is equal to the first focus position, and the second focal length and the second F-number in the target image capturing parameters are equal to the first focal length and the first F-number in the reference image capturing parameters. The definition calculation unit,
7. The image processing system according to claim 1, further comprising a storage unit in which the subject image resolution and the reference captured image imaging parameters are written and stored for each type of the imaging device. The definition calculation unit,
The image processing system according to any one of claims 1 to 7, characterized in that the subject image is compared with an adjusted subject CG image obtained by filtering the subject CG image with a weighted average filter, and a first filter radius of the weighted average filter that equalizes the difference between the subject image and the adjusted subject CG image is calculated as the subject image resolution. The definition calculation unit,
The image processing system according to claim 8 , further comprising: determining a second filter radius, which is the virtual object image definition, from the first filter radius in accordance with the correspondence relationship, and adjusting the definition of the virtual object image. a subject information estimation step in which a subject information estimation unit estimates subject information including a position and an orientation of the subject relative to an imaging device based on a subject image of the subject in a reference captured image captured by the imaging device;
a subject CG image drawing step in which a subject CG image drawing unit draws a subject CG image that is a CG image of the subject based on the subject information;
a virtual object placement position acquisition step in which a virtual object placement position acquisition unit acquires a placement position at which a virtual object image is to be placed in a target captured image captured by the imaging device;
a resolution calculation process in which a resolution calculation unit compares the subject image with the subject CG image to acquire a subject image resolution, which is the resolution of the subject image, and also uses a correspondence relationship to obtain a virtual object image resolution, which is the resolution of a virtual object image to be synthesized with the target captured image, from the subject image resolution, reference captured image imaging parameters when the reference captured image was captured by the imaging device, a first distance between the subject and the imaging device, and a second distance, which is a virtual distance between the imaging device and the virtual object of the virtual object image when the virtual object image is synthesized with the target captured image, to acquire the virtual object image resolution corresponding to the second distance. Computer,
a subject information estimating means for estimating subject information including a position and an orientation of the subject relative to an imaging device based on a subject image of the subject in a reference captured image captured by the imaging device;
a subject CG image drawing means for drawing a subject CG image which is a CG image of the subject based on the subject information;
a virtual object placement position acquisition means for acquiring a placement position for arranging a virtual object image in a target captured image captured by the imaging device;
a program that functions as a resolution calculation means for comparing the subject image with the subject CG image to acquire a subject image resolution that is the resolution of the subject image, and also for using a correspondence relationship to obtain a virtual object image resolution that is the resolution of a virtual object image to be composited into the target captured image from the subject image resolution, reference captured image imaging parameters when the reference captured image was captured by the imaging device, a first distance between the subject and the imaging device, and a second distance that is a virtual distance between the imaging device and the virtual object of the virtual object image when the virtual object image is composited into the target captured image, to acquire the virtual object image resolution that corresponds to the second distance.

Images