JP4579730B2 - Image processing apparatus, image processing method and program thereof - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program thereof used in an information presentation system that presents information using a projection apparatus such as a projector.

  Conventionally, as a means for sharing information among a plurality of people, a method of projecting information on a real object has been proposed (see, for example, Patent Document 1). According to this, not only in face-to-face communication, but also when transmitting information between remote locations, such as a remote work support system, It can be projected with a projector. In this way, information can be easily and accurately transmitted by visually presenting information on the real space.

JP 2001-211132 A

  However, when the target object in the real space is photographed in a state where information is superimposed on the target object existing in the real space and projected as in the conventional information presentation method, the projected information is displayed together with the target object. Will appear in the image. For this reason, it is difficult to distinguish between an object existing in real space and the projected information from the captured image. For this reason, for example, when the projected information is similar to the object, the object may not be correctly recognized from the captured image by image processing.

  The present invention has been made paying attention to the above circumstances, and the object of the present invention is to accurately recognize an object in the real space, even when information is projected on the object existing in the real space, and to An object of the present invention is to provide an image processing apparatus, an image processing method, and a program thereof that can reduce errors in processing.

In order to achieve the above object, an image processing apparatus, an image processing method, and a program therefor according to the present invention photograph an object existing in a space and a post-projection image projected by the projection device in the space with a photographing device. Means for acquiring a captured image obtained by the above, means for acquiring a pre-projection image projected by the projection device, and means for subtracting the pre-projection image from the acquired captured image.
With this configuration, even when information is projected onto an object existing in the real space, it is possible to accurately recognize the object in the real space and reduce errors in image processing.

The present invention is also characterized by having the following various configurations.
According to a first configuration, there is provided means for acquiring a captured image obtained by photographing an object existing in a space and a post-projection image projected on the space by a projection device, and the projection device. Means for acquiring the pre-projection image, means for estimating the post-projection image based on the acquired pre-projection image, and means for subtracting the estimated post-projection image from the acquired photographed image Is provided.
In this way, a post-shooting image projected by a projection device such as a projector is estimated, and the estimated post-shooting image is subtracted from the picked-up image, so that an object in real space can be recognized more accurately.

In the second configuration, the means for estimating the post-projection image is based on the means for detecting information representing the color characteristics of the projection device, the information representing the detected color characteristics, and the pre-projection image. In addition, the post-projection image is estimated.
In this way, the post-capture image is accurately estimated without being influenced by the color characteristics of the projection apparatus, so that an object in real space can be reliably recognized.

In the third configuration, the means for detecting the information representing the color characteristics includes means for obtaining a non-projected image obtained by photographing the space by the photographing apparatus in a state where the projection apparatus does not perform projection, and three primary colors of light. Means for projecting each pattern image corresponding to each constituent color on the projection device; means for obtaining each pattern photographed image obtained by photographing each projected pattern image by the photographing device; Means for detecting information representing color characteristics corresponding to the three constituent colors from the difference between each pattern captured image and the non-projected image.
If comprised in this way, the color characteristic which a projector has can be detected correctly. Thereby, it is possible to reliably recognize an object in real space without being influenced by the color characteristics of the projection apparatus.

According to a fourth configuration, the means for detecting the information representing the color characteristics includes means for obtaining a non-projected image obtained by photographing the space with the photographing apparatus in a state where the projection apparatus does not perform projection, and a white light pattern. Means for projecting an image on the projection device, means for obtaining a white light pattern photographed image obtained by photographing the projected white light pattern image with the photographing device, and the obtained white light pattern photographed image, Information representing color characteristics corresponding to the three primary colors is detected from the difference from the non-projected image.
By using the white light pattern, it is possible to obtain information representing the color characteristics of the three primary colors without requiring time and labor for one projection and photographing.

According to a fifth configuration, the means for estimating the post-projection image includes means for detecting information indicating a luminance change characteristic of the projection device, information indicating the detected luminance change characteristic, and the pre-projection image. Based on the above, the post-projection image is estimated.
In this way, the post-capture image is accurately estimated without being affected by the luminance change characteristic of the projection device, so that an object in real space can be reliably recognized.

  In short, according to the present invention, even when information is projected onto an object existing in the real space, the image processing apparatus that can accurately recognize the object in the real space and reduce errors in the image processing. In addition, an image processing method and a program thereof can be provided.

FIG. 1 is a schematic configuration diagram showing an information presentation system including an image processing apparatus according to the present invention.
This information presentation system includes an image processing apparatus 1, a camera 2, and a projector 3. The camera 2 and the projector 3 are connected to the image processing apparatus 1 respectively. The camera 2 is constituted by a digital camera or a video camera having a 3CCD (Charge Coupled Device) solid-state imaging device. The camera 2 and the projector 3 are fixed so that the shooting range of the camera 2 and the projection range of the projector 3 coincide as shown in FIG. In this state, when an image captured by the camera 2 is output to the projector 3 as it is, the image captured by the object 5 is projected onto the object 5 existing in the real space. The projected image 51 is an image that represents information related to the object 5 projected by the projector 3.

FIG. 2 is a block diagram illustrating a functional configuration of the image processing apparatus 1.
The image processing apparatus 1 includes a central processing unit (CPU) 11 as a main control unit. A data memory 13 and a program memory 14 are connected to the CPU 11 via a bus 12, and further a camera input interface (camera input I / F) 15, a projector output interface (projector output I / F) 16, and an input / output interface. (Input / output I / F) 17 is connected.

  The camera input I / F 15 inputs an image photographed by the camera 2 under the control of the CPU 11. The projector output I / F 16 is a connection interface with the projector 3 and outputs a projection source image to the projector 3 under the control of the CPU 11. The input / output I / F 17 is connected to an input device 18 such as a mouse and a keyboard and a display device 19 such as a liquid crystal display (LCD). The input device 18 is a user interface for inputting image data, video data, text data, or the like for projecting the projection image 51. The display device 19 displays an image captured by the camera 2, an image subjected to image processing described later, and the like.

The data memory 13 is composed of, for example, a RAM or a hard disk, and stores a correction parameter storage area 13a, a projection source image storage area 13b, an estimated reflected image storage area 13c, and a difference image storage area 13d.
In the correction parameter storage area 13a, correction parameters representing the color characteristics of the projector 3 are stored for each pixel for each of the R, G, and B planes. In the projection source image storage area 13b, a projection source image that is the basis of the projection image 51 projected by the projector 3 is stored. As described above, the projection source image is created based on image data, video data, text data, or the like input from the input device 18. The created projection source image is output to the projector 3 by the projector output I / F 16, and the projector 3 projects the projection image 51 based on the projection source image.

  The estimated reflected image storage area 13c stores an estimated reflected image obtained by correcting the projection source image based on the correction parameter stored in the correction parameter storage area 13a. The difference image storage area 13d stores a difference image that is a difference between the corrected estimated reflected image and the image taken by the camera 2.

  The program memory 14 stores a correction parameter calculation program 14a, an estimated reflected image creation program 14b, and a difference image extraction program 14c as control programs for realizing the present invention. The correction parameter calculation program 14a has a function of calculating a correction parameter representing the color characteristic by projection of the projector 3 and storing it in the correction parameter storage area 13a described above.

The estimated reflected image creation program 14b reads the correction parameter stored in the correction parameter storage area 13a and the projection source image stored in the projection source image storage area 13b, respectively, and corrects the read projection source image. An estimated reflected image is created by correcting with parameters. The created estimated reflected image is stored in the estimated reflected image storage area 13c.
The difference image extraction program 14c extracts the difference by comparing the created estimated reflected image and the image taken by the camera 2, and stores the extracted difference image in the difference image storage area 13d. .

  Next, the operation of the image processing apparatus 1 configured as described above will be described. FIG. 3 is a flowchart showing the procedure and contents of the correction parameter calculation process of the image processing apparatus 1. As described above, the camera 2 is composed of a 3CCD camera having three planes R, G, and B. This correction parameter calculation process is performed for each of the R, G, and B planes. However, since this process is the same, the R plane will be described as an example here.

  First, in step S3a, the CPU 11 of the image processing apparatus 1 captures an image captured by the camera 2 in a state in which projection by the projector 3 is not performed (hereinafter referred to as “no projection”) using the camera input I / F 15 without projection. Import as an image. Next, in step S3b, an image captured by the camera 2 in a state in which the correction pattern (in the case of the R plane, the maximum luminance image of all pixels red) is projected by the projector 3 is used as the correction pattern captured image. Capture by F15.

  Subsequently, the CPU 11 of the image processing apparatus 1 executes the correction parameter calculation program 14a, and repeats the processes of steps S3c to S3f for each pixel. That is, in step S3d, the difference in luminance of each pixel between the captured correction pattern captured image and the non-projection captured image is calculated. In step S3e, the calculated luminance difference is stored in the correction parameter storage area 13a as a correction parameter.

  Similarly, the same processing is performed for the G plane and the B plane, and the correction parameters for the R, G, and B planes are stored in the correction parameter storage area 13a. The correction pattern used in step S3b uses the maximum luminance image of all pixels green for the G plane and the maximum luminance image of all pixels blue for the B plane.

  Next, an estimated reflected image creation operation in the image processing apparatus 1 will be described using the stored correction parameters. FIG. 4 is a flowchart showing the procedure of the estimated reflected image creating process of the image processing apparatus 1 and its contents. Here, the estimated reflected image is an image in which the projection source image is projected by the projector 3, that is, an image obtained by estimating the projection image 51 projected in the real space.

The CPU 11 of the image processing apparatus 1 executes the estimated reflected image creation program 14b and reads the projection source image stored in the projection source image storage area 13b. Then, the correction process of steps S4b to S4f is repeated for each pixel x. In step S4c, the R plane is corrected as follows. First, the correction parameter R (x) of the pixel x of the R plane is read from the correction parameter storage area 13a. Also, assuming that the maximum luminance value of the R plane is Rmax (usually 255) and the luminance value of the pixel x of the projection source image is R0 (x), the luminance value R1 (x) of the pixel x of the R plane in the estimated reflected image Is obtained as follows.
R1 (x) = R (x) / Rmax × R0 (x)
Similarly, for the G plane in step S4d, assuming that the correction parameter of the pixel x of the G plane is G (x), the maximum luminance value is Gmax, and the luminance value of the pixel x of the projection source image is G0 (x), the estimated reflection The luminance value G1 (x) of the pixel x of the G plane in the image is
G1 (x) = G (x) / Gmax × G0 (x)
Is required.

Similarly, assuming that the correction parameter for the pixel x of the B plane is B (x), the maximum luminance value is Bmax, and the luminance value of the pixel x of the projection source image is B0 (x) for the B plane in step S4e. The luminance value B1 (x) of the pixel x of the B plane in the reflected image is
B1 (x) = B (x) / Bmax × B0 (x)
Is required.

  After performing the correction processing of step S4b to step S4f for each pixel, the process proceeds to step S4g, and the estimated reflected image is obtained by synthesizing the calculated pixel values of the R, G, and B planes for each pixel. The created estimated reflected image is stored in the estimated reflected image storage area 13c.

Next, the difference image extraction operation of the image processing apparatus 1 will be described using the stored estimated reflected image. FIG. 5 is a flowchart showing the procedure and contents of the differential image extraction process of the image processing apparatus 1.
The CPU 11 of the image processing apparatus 1 executes the differential image extraction program 14c, and captures an image obtained by photographing the projection image 51 by the projector 3 together with the object 5 by the camera input I / F 15 in step S5a. In step S5b, an estimated reflected image is read from the estimated reflected image storage area 13c. In step S5c, a difference image is extracted by comparing the captured captured image with the read estimated reflected image. In step S5d, the extracted difference image is stored in the difference image storage area 13d.

  In this way, the differential image stored in the differential image storage area 13d is output to the display device 19 via the input / output I / F 17, for example. Specifically, for example, as shown in FIG. 6, when the image captured by the camera 2 is an image including the object 5 in real space and the projection image 51, the image processing apparatus 1 uses the projection source image Is estimated using the correction parameter, and a difference image is created by extracting the difference between the image captured by the camera 2 and the estimated reflection image. As described above, the difference image is an image including only the target 5 in the real space.

  In addition, the positional relationship of each pixel of the projection image by the projector 3 and the picked-up image by the camera 2 shall be acquired beforehand. For example, by performing calibration so that the angles of view of the camera 2 and the projector 3 coincide with the optical axis, it is possible to obtain a correspondence relationship between each pixel of the projected image and the captured image of the camera 2. In addition to this, a calibration pattern such as a colored lattice image is projected by the projector 3, and the correspondence between the position of the lattice on the captured image by the camera 2 and the position of the lattice on the projected image is obtained. It can also be done. For the R, G, and B planes, the calibration may be performed for each plane.

  As described above, in this embodiment, the CPU 11 of the image processing apparatus 1 executes the correction parameter calculation program 14a, and the captured image of the camera 2 when the projector 3 is not projected and the R, G, B planes. The difference in luminance of each pixel is calculated based on the captured image of the camera 2 at the time of projecting the correction pattern, which is the maximum luminance image, and stored as a correction parameter. Further, the CPU 11 creates an estimated reflected image obtained by correcting the projection source image based on the correction parameter, and stores the estimated reflected image in the estimated reflected image storage area 13c. Then, the CPU 11 extracts the difference between the photographed image of the camera 2 and the created estimated reflected image by the difference image extraction program 14c, and stores the extracted difference image in the difference image storage area 13d. Yes.

  Therefore, according to this embodiment, it is possible to provide an image of only the object when the object is photographed in a state where information is superimposed on the object existing in the real space and projected. . For example, as shown in FIG. 6, when a captured image of a camera includes both a human hand in a real space and a human hand in a projection image, an image processing algorithm is used to extract a human image from the captured image. Suppose you recognize your hand. In this case, the human hand in the real space and the human hand in the projection image may not be distinguished from each other and may be erroneously recognized. However, in the present invention, since an image of only the object 5 existing in the real space is displayed, it is possible to reduce the occurrence of erroneous recognition in image processing.

  The present invention is not limited to the above embodiment. In the above-described embodiment, an estimated reflected image in which the projected image 51 is estimated based on the projection source image is created, and this estimated reflected image is subtracted from the captured image of the camera 2. The pre-projection image may be subtracted from the photographed image of the camera 2 based on the projection source image. This is particularly effective for a projector having excellent color reproducibility, and the processing for calculating correction parameters and the processing for creating an estimated reflected image can be omitted, so that the processing load on the CPU can be reduced.

  In the correction parameter calculation process, the maximum luminance image of white light may be used as the correction pattern. In this way, since the correction pattern is projected and the captured image is captured by the camera 2 only once, it is possible to save the trouble of switching and projecting the correction pattern for each of the R, G, and B planes.

  In the above-described embodiment, the R, G, and B color characteristics of the projector 3 are detected. However, it is also possible to detect a change characteristic depending on luminance (brightness). In this way, an object in real space can be accurately recognized without being influenced by the luminance change characteristic of the projection apparatus. Further, not only the maximum luminance image of each plane but also a correction pattern in which the luminance is changed may be used to detect a change in color characteristics due to brightness.

  Further, in the correction parameter calculation process, a correction parameter may be obtained for each pixel as in the above-described embodiment. You may make it calculate only about a big part. In this way, the processing speed in calculating the correction parameter is improved and the memory capacity required for the correction parameter storage area 13a is saved.

In addition, the gist of the present invention is also applied to the configuration of the image processing apparatus 1, the items and configurations of each storage area stored in the data memory 13, and the processing procedure and processing contents of each program stored in the program memory 14. Various modifications can be made without departing from the scope.
In short, the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying its constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, you may combine the component covering different embodiment suitably.

BRIEF DESCRIPTION OF THE DRAWINGS Schematic which shows an information presentation system provided with the image processing apparatus concerning 1st Embodiment of this invention. The block diagram which shows the function structure of the image processing apparatus shown in FIG. 3 is a flowchart showing a procedure and details of correction parameter calculation processing of the image processing apparatus shown in FIG. The flowchart which shows the procedure and the content of the presumed reflection image creation process of the image processing apparatus shown in FIG. The flowchart which shows the procedure and the content of the difference image extraction process of the image processing apparatus shown in FIG. The figure which shows the mode of the difference image extraction process by the image processing apparatus shown in FIG.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Image processing apparatus, 2 ... Camera 2 ... Projector, 5 ... Object, 51 ... Projection image, 11 ... CPU, 12 ... Bus, 13 ... Data memory, 13a ... Correction parameter storage area, 13b ... Projection original image storage area , 13c ... Estimated reflected image storage area, 13d ... Differential image storage area, 14 ... Program memory, 14a ... Correction parameter calculation program, 14b ... Estimated reflected image creation program, 14c ... Differential image extraction program, 15 ... Camera input I / F, 16 ... Projector output I / F, 17 ... Input / output I / F, 18 ... Input device, 19 ... Display device.

Claims (3)

Means for acquiring a photographed image obtained by photographing an object existing in space and a post-projection image projected by the projection device in the space with a photographing device;
Means for storing a pre-projection image projected by the projection device;
Means for detecting information representing color characteristics of the projection device;
Means for estimating the post-projection image based on the information representing the detected color characteristics and the pre-projection image;
Means for subtracting the estimated post-projection image from the acquired photographed image,
The means for detecting information representing the color characteristic is:
Means for acquiring a non-projected image obtained by photographing the space by the photographing device in a state where the projection device does not perform projection;
Means for projecting a pattern image of white light on the projection device;
Means for acquiring a white light pattern photographed image obtained by photographing the projected white light pattern image by the photographing device;
An image processing apparatus comprising: means for detecting information representing color characteristics corresponding to each of the three primary colors from a difference between the acquired white light pattern captured image and the non-projected image. . Obtaining a captured image obtained by photographing an object existing in space and a post-projection image projected by the projection device in the space with a photographing device;
Storing a pre-projection image projected by the projection device;
Detecting information representing color characteristics of the projection device;
Estimating the post-projection image based on the information representing the detected color characteristics and the pre-projection image;
Estimating the post-projection image based on the acquired pre-projection image;
Subtracting the estimated post-projection image from the acquired captured image;
Have
The step of detecting information representing the color characteristic includes:
Obtaining a non-projected image obtained by photographing the space with the photographing device in a state in which the projection device does not perform projection;
Projecting a white light pattern image onto the projection device;
Obtaining a white light pattern photographed image obtained by photographing the projected white light pattern image with the photographing device;
Detecting information representing color characteristics corresponding to the three primary colors from the difference between the acquired white light pattern captured image and the non-projected image;
An image processing method comprising: An image processing program for causing a computer to function as each means constituting the image processing apparatus according to claim 1.

Images