JP6464968B2 - Projection control apparatus, program, and projection system - Google Patents

Description

  The present invention relates to a technical field of a projection system that projects an image on a projection surface.

  2. Description of the Related Art Conventionally, there has been known an apparatus that projects an image in which a region projected on a person in a projection image is masked in order to reduce a visual stimulus to a person existing within the projection range of the projection image projected from the projector. For example, the projection display device disclosed in Patent Document 1 compares an original image to be projected with a projection image captured by a camera, detects a different area as a difference area, and detects the difference area as a black signal. A masked image is generated, and the generated image is projected on a screen.

JP 2004-254145 A

  By the way, when a person who is within the projection range of the projection image and the projection plane and the projection image projected onto the person are captured and recorded by the camera, the person is seen in front of the projection plane when viewed from the camera. Depending on the intensity of light emitted from the projector, an unnatural image with a sense of incongruity may be recorded as compared to an image obtained by capturing a background and a person with a camera. Examples of such an unnatural image with a sense of incongruity include an image in which a person's region has been blown out, an image in which the color of a person's clothes has changed from the primary color, and an image in which a person's shadow is reflected. Furthermore, depending on the positional relationship between the person and the projector, the light emitted from the projector may not strike the person equally. In this case, even in the human area, for example, an image in which a whiteout area and a nonexistent area are mixed, or an image in which a color change due to the light intensity and a nonexistent area are mixed is recorded. It may be done.

  The present invention has been made in view of the above points, and is a case where an object (for example, a person) existing within a projection range of a projection image and a projection surface and a projection image projected on the object are imaged. There are provided a projection control device, a program, and a projection system capable of capturing and recording a natural image without any sense of incongruity.

In order to solve the above-mentioned problem, the invention according to claim 1 is characterized in that a first storage means for storing an image to be projected onto a projection plane by a projection device, and a person from a region of an object existing within the projection range of the image. Detecting means for detecting a face area of the object, and corresponding to an area where the reflectance of the light emitted from the projection device is relatively high in the area of the object existing within the projection range of the image with respect to the image Setting means for setting a gradation of the area of the image to be lower than a gradation of the area of the image corresponding to the area having a relatively low reflectance, the setting of the face detected by the detection means In the region of the image corresponding to the region, a setting unit that sets a gray gradation having a higher gradation than the surroundings of the region, and the image including the region in which the gradation is set by the setting unit are projected images. Projected on the projection device as The captured image captured by the imaging device using the control means, the object existing within the projection range of the projection image, and the projection image projected onto the projection plane and the object by the projection device as a captured image And a second storage means for storing the captured image acquired by the acquisition means.

  According to a second aspect of the present invention, in the projection control apparatus according to the first aspect, the setting means sets a gray gradation in which the gradation increases in order from the high reflectance region to the low reflectance region. It is characterized by that.

  According to a third aspect of the present invention, in the projection control apparatus according to the second aspect, a gradation change table in which gray gradations having different gradations according to the reflectance are defined in a coordinate system common to the image is provided. The image forming apparatus further includes a creating unit that creates, and according to the tone change table, the setting unit sets a gray gradation in which the gradation increases in order from the high reflectance region to the low reflectance region. To do.

According to a fourth aspect of the present invention, the step of storing in the storage means an image to be projected onto the projection plane by the projection device, and the step of detecting a human face region from the region of the object existing within the projection range of the image. And the gradation of the region of the image corresponding to a region where the reflectance of light emitted from the projection device is relatively high within the region of the object existing within the projection range of the image. Setting a lower level than the gradation of the image area corresponding to the area having a relatively low reflectance, and the area of the image corresponding to the detected face area is the area. A step of setting a gray gradation higher in gradation than the surroundings, a step of projecting the image including the region in which the gradation is set to the projection device as a projection image, and within a projection range of the projection image Existing objects Acquiring the captured image captured by the imaging device using the projection image projected from the projection device and the projection image projected onto the object as the captured image; and storing the acquired captured image as the storage unit. And storing in the computer.

The invention according to claim 5 is a projection system comprising: a projection device that projects an image onto a projection plane; an imaging device that captures the projected image; and a projection control device that controls the projection. The projection control device includes: a first storage unit that stores an image to be projected onto a projection plane by the projection device; and a detection unit that detects a human face region from a target region existing within a projection range of the image. , For the image, the gradation of the region of the image corresponding to the region where the reflectance of light emitted from the projection device is relatively high within the region of the object existing within the projection range of the image, A setting means for setting the gradation of the image area corresponding to the area having a relatively low reflectance , wherein the area of the image corresponding to the face area detected by the detection means is , Floors around the area Comprising setting means sets the high gray level, and a control means for outputting a projection command to project the image including the area where the gradation is set as a projection image to the projection device by the setting unit The projection device includes projection means for projecting the projection image onto the projection surface and the object in response to the projection command from the projection control device, and the imaging device is within a projection range of the projection image. An imaging unit that captures, as a captured image, an object that is present on the projection surface and the projection image projected onto the projection plane and the object from the projection device, and the projection control device is captured by the imaging device An acquisition unit that acquires the captured image, and a second storage unit that stores the captured image acquired by the acquisition unit.

According to the inventions described in claims 1, 4 , and 5 , even when the object existing within the projection range of the projection image and the projection image projected onto the projection surface and the object are imaged, there is a sense of incongruity. free can be recorded by imaging a natural image, Ru can be part of a face of a person is recorded by imaging a better natural look image.

  According to the second and third aspects of the invention, it is possible to capture and record a smoother natural image within an object existing within the projection range of the projection image.

(A) is a figure which shows the example of a schematic structure of the projection system S of this embodiment. (B) is a figure showing an example of the positional relationship of the apparatus arrange | positioned in the room seen from the upper direction of the room where the projection system S is utilized. (A) is a flowchart which shows an example of the instruction | indication reception process in a projection control process. FIG. 6B is a diagram illustrating an example of an instruction reception screen displayed on the display 5. It is a flowchart which shows an example of a linear complementation LUT creation process. It is a conceptual diagram which shows the example which produces | generates the synthesized image P3 by projecting and imaging separately the monochrome pattern image PO1 and the black-and-white pattern image OP2. Projector coordinates _{(x p, y p)} is a conceptual diagram showing the correspondence give examples of the camera coordinates _{(x c, y c).} It is a conceptual diagram which shows an example of a linear complement LUT. It is a flowchart which shows an example of a projection and an imaging process. (A) is a flowchart which shows an example of a difference image generation process. (B) is a flowchart showing an example of a mask correction image generation process. It is a figure which shows an example of the background image B, the captured image C1, the difference image D, and the mask correction image M. It is a conceptual diagram which shows an example of LUT for whitening modes.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[1. Outline of Projection System Configuration and Operation]
First, with reference to FIG. 1 etc., the structure and operation | movement outline | summary of the projection system S of this embodiment are demonstrated. As shown in FIG. 1A, the projection system S includes a video playback device 1, a projector 2, a camera 3, an operating device 4, a display 5, a projection control device 6, and the like. The video reproduction device 1 and the projector 2 are examples of a projection device. The camera 3 is an example of an imaging device. The projection system S is used in a room in an amusement facility such as a karaoke store. The video playback device 1, the camera 3, the operating device 4, and the display 5 are connected to the projection control device 6 via a wired or wireless connection. The projector 2 is connected to the video playback device 1 via a wired or wireless connection.

  The video playback device 1 is a device that plays back an image (video). There are still images and moving images. A moving image is displayed, for example, by switching image frames of a plurality of still images in time series. For example, the video reproduction device 1 reproduces an image instructed from the projection control device 6 and outputs the image signal to the projector 2. Image data of an image to be reproduced is stored in a storage unit (not shown) of the video reproduction device 1. Alternatively, the video reproduction device 1 may acquire image data of an image to be reproduced from the storage unit of the projection control device 6. The projector 2 outputs an image by emitting light from the light emitting unit according to the image signal reproduced and output by the video reproduction device 1. As a result, the projector 2 projects an image (hereinafter referred to as “projection image”) as a projection unit on a projection plane within a projection range that falls within the projection angle. Examples of the projection image include an original image and a mask correction image M. The original image is an image provided from an image provider, for example. Examples of the original image include a promotion image of an artist and an advertisement image of a company or a product. The mask correction image M is an image obtained by performing mask correction processing on a partial area of the original image. In this mask correction process, an area (one in the original image) corresponding to an area where the reflectance of light emitted from the projector 2 is relatively high within the area of an object (for example, a person) existing within the projection range of the original image. The gradation of the partial area) is set low, and the gradation of the area corresponding to the area where the reflectance is relatively low (other partial areas in the original image) is set high. Details of the mask correction process will be described later.

  In the example of FIG. 1B, the projector 2 is installed on the upper part (for example, the ceiling) of the room, and projects an image on a projection plane including a part of the wall surface a, a part of the wall surface b, and a person. The ceiling surface or floor surface of the room may be a projection surface. The projection surface may be a curved surface. A plurality of projectors 2 may be arranged in the room. In this case, each projector 2 is arranged at a different position, and projects an image from each position onto a projection plane within a projection range that falls within each projection angle. At this time, each projector 2 may project an image by overlapping a part of each projection range. When a plurality of projectors 2 are arranged, a plurality of projectors 2 may be connected to one video reproduction device 1, or one projector 2 may be connected to each of the plurality of video reproduction devices 1. Good. Further, the video reproduction device 1 may be incorporated in the projection control device 6.

  The camera 3 includes imaging means for imaging an imaging range that falls within the angle of view. In the example of FIG. 1B, the camera 3 captures a person existing within the projection range of the projection image and the projection surface and the projection image projected onto the person by the projector 2 as captured images. That is, the imaging range illustrated in FIG. 1B is captured. The captured image captured by the camera 3 is stored in the storage unit of the projection control device 6. As described above, there are two types of projection images: the original image and the mask correction image M. For this reason, at least a captured image when the original image is projected (hereinafter referred to as “captured image C1”) and a captured image when the mask correction image M is projected (hereinafter referred to as “captured image C2”) are stored. Will be stored in the department. As the camera 3, for example, an RGB camera is used.

  The operating device 4 receives an operation instruction from the user, and outputs an instruction signal indicating the operation instruction to the projection control device 6. An instruction acceptance screen is displayed on the display 5. On the instruction reception screen, an LUT (Look Up Table) creation start button, a normal mode setting button, a whitening mode setting button, a projection / imaging start button, and the like are displayed. The LUT creation start button is a button for instructing the projection control device 6 to create a linear interpolation LUT used in the mask correction process. Details of the linear interpolation LUT will be described later. When the user operates the operation device 4 to specify the LUT creation start button, an instruction signal indicating a linear interpolation LUT creation instruction is output to the projection control device 6. The normal mode setting button is a button for instructing the projection control apparatus 6 to set the normal mode. Here, the normal mode is an example of an execution mode of mask correction processing, and is a mode in which the mask correction processing is executed using a linear interpolation LUT. When the user operates the operation device 4 to specify a normal mode setting button, an instruction signal indicating a normal mode setting instruction is output to the projection control device 6. The whitening mode setting button is a button for instructing the projection control device 6 to set the whitening mode. Here, the whitening mode is an example of an execution mode of mask correction processing, and a mode in which the mask correction processing is executed using an LUT created based on a linear interpolation LUT (hereinafter referred to as “LUT for whitening mode”). It is. When the user operates the operation device 4 to specify a whitening mode setting button, an instruction signal indicating a whitening mode setting instruction is output to the projection control device 6. Details of the whitening mode LUT will be described later. The projection / imaging start button is a button for instructing the projection control device 6 to start projecting an image and start imaging. When the user operates the operation device 4 to specify a projection / imaging start button, an instruction signal indicating an image projection and imaging start instruction is output to the projection control device 6. The operation device 4 and the display 5 may be an integrated touch panel display.

  The projection control device 6 is a device that performs image projection control and the like. As shown in FIG. 1A, the projection control device 6 includes IFs 61a to 61d, a storage unit 62, and a control unit 63. The IFs 61 a to 61 d, the storage unit 62, and the control unit 63 are connected to the bus 64. The IF 61 a is an interface between the video reproduction device 1 and the projection control device 6. The IF 61 b is an interface between the camera 3 and the projection control device 6. The IF 61 c is an interface between the operation device 4 and the projection control device 6. The IF 61 d is an interface between the display 5 and the projection control device 6.

  The storage unit 62 is configured by, for example, a hard disk drive. The storage unit 62 is an example of a first storage unit and a second storage unit. The storage unit 62 stores an OS (Operating System), the program of the present invention, and the like. The program of the present invention is a program for causing a CPU to execute a control process described later. Further, the storage unit 62 stores image data of an original image projected by the projector 2. In addition, the storage unit 62 stores image data of the captured image D1 captured by the camera 3, image data of the captured image D2 captured by the camera 3, and the like. The control unit 63 includes a CPU (Central Processing Unit) as a computer, a ROM (Read Only Memory), a RAM (Random Access Memory), and the like. The control unit 63 is an example of a setting unit, a control unit, an acquisition unit, a creation unit, and a detection unit of the present invention. The control unit 63 executes projection control processing according to the program of the present invention.

[2. Projection Control Processing of Projection Control Device 6]
Next, the projection control process of the projection control apparatus 6 of the present embodiment will be described with reference to FIG. The process shown in FIG. 2A is started, for example, when the projection control apparatus 6 is turned on. When the process shown in FIG. 2A is started, the control unit 63 displays, for example, an instruction reception screen shown in FIG. 2B on the display 5 (step S1), and waits for an input of an operation instruction from the user. It becomes a state. Next, the control unit 63 determines whether or not there is a linear interpolation LUT creation instruction (step S2). For example, when the user operates the operation device 4 and specifies the LUT creation start button 51 displayed on the instruction reception screen, an instruction signal indicating a linear complementary LUT creation instruction is input to the projection control device 6. The unit 63 determines that there is a linear interpolation LUT creation instruction. When it is determined that the linear interpolation LUT creation instruction has been issued (step S2: YES), the control unit 63 proceeds to step S3. On the other hand, when the control unit 63 determines that there is no linear interpolation LUT creation instruction (step S2: NO), the control unit 63 proceeds to step S4.

  In step S3, the control unit 63 executes a linear interpolation LUT creation process. Here, an example of the linear interpolation LUT creation process will be described with reference to FIG. In step S31 illustrated in FIG. 3, the control unit 63 transmits a projection command for the monochrome pattern image OP1 to the video reproduction device 1. As shown in FIG. 4A, the black-and-white pattern image OP1 is an image that is composed of gray gradations and in which a white region and a black region are alternately repeated by a predetermined number. In the example of FIG. 4A, the number of white areas and black areas that are alternately repeated is four, but may be five or more. The gray gradation of the white area in the monochrome pattern image PO1 is set to “255”, for example, and the gray gradation of the black area in the monochrome pattern image OP1 is set to “0”, for example. The image data of the monochrome pattern image OP1 may be stored in advance in the storage unit of the video reproduction device 1, or may be transmitted to the video reproduction device 1 together with a projection command for the monochrome pattern image OP1. When receiving the projection command for the monochrome pattern image OP 1 from the projection control device 6, the video reproduction device 1 reproduces the monochrome pattern image OP 1 and outputs the image signal to the projector 2. The projector 2 projects the monochrome pattern image OP1 as a projection image on the projection plane in the projection range according to the reproduced and output image signal. There is no person in the projection range at this time.

  Next, the control unit 63 transmits an imaging command for the monochrome pattern image OP1 to the camera 3 (step S32). When the camera 3 receives an imaging command for the monochrome pattern image OP1 from the projection control device 6, the monochrome pattern image OP1 within the imaging range that falls within the angle of view of the camera 3 is taken as a captured image P1 as shown in FIG. The captured image P1 is captured and transmitted to the projection control device 6. In the captured image P1 shown in FIG. 4B, the gray gradation decreases (that is, the brightness decreases) as the white region has a larger distance in the y-axis direction from the origin (0, 0). This is because the larger the distance in the y-axis direction from the origin (0, 0), the farther from the installation position of the projector 2. For the black area, the gray gradation changes as the distance from the installation position of the projector 2 increases. Next, the control unit 63 acquires (receives) and records the captured image P1 from the camera 3 (step S33). Thereby, for example, the image data of the captured image P1 and the identification ID of the captured image P1 (for example, generated at the time of image acquisition) are associated with each other and stored in the storage unit 62.

By the way, the coordinates in the projection image output from the projector 2 (hereinafter referred to as “projector coordinates”) may not match the coordinates in the captured image captured by the camera 3 (hereinafter referred to as “camera coordinates”). is there. For example, when the monochrome pattern image OP1 shown in FIG. 5B is projected across the wall surface a and the wall surface b as shown in FIG. 5A, the monochrome pattern image OP1 is captured by the camera 3. In addition, the contours of the eyelids that are originally linear (that is, the contours of the white region and the black region) are captured in a refracted state as in the captured image P0 illustrated in FIG. In such a case, the pre-processing for associating the projector coordinates (x _p , y _p ) with the camera coordinates (x _c , y _c ) is performed, for example, before the execution of the linear interpolation LUT creation process. Then, a correspondence table for associating the projector coordinates (x _p , y _p ) with the camera coordinates (x _c , y _c ) is created. The correspondence between the projector coordinates (x _p , y _p ) and the camera coordinates (x _c , y _c ) is, for example, “RECENT PROGRESS IN CODED STRUCTURED LIGHT AS TECHNIQUE TO SOLVE THE CORRESPONDENCE PROBLEM: A SURVEY ”Pattern (Recognition, Vol. 31, No. 7, pp. 963-982, 1998) may be used. Thereby, for example, the coordinates (x _p0 , y _p0 ) in the monochrome pattern image OP1 shown in FIG. 5B are associated with the coordinates (x _c0 , y _c0 ) in the captured image P0 shown in FIG. . When acquiring the captured image P1 from the camera 3, the control unit 63 in the case of using the association table obtains the coordinates (x _c , y _c ) of the camera in the captured image P1 according to the association table created in advance. The coordinates (x _p , y _p ) are converted into a captured image P1 having the converted coordinates (step S33).

  Next, the control unit 63 transmits a projection command for the black and white pattern image OP2 to the video reproduction device 1 (step S34). As shown in FIG. 4C, the black-and-white pattern image OP2 is a black-and-white reversal pattern image obtained by reversing the white area and black area of the black-and-white pattern image OP1 shown in FIG. A black region and a white region are alternately repeated by a predetermined number. The gray gradation of the black area in the black and white pattern image OP2 shown in FIG. 4C is set to, for example, “0”, and the gray gradation of the white area in the black and white pattern image OP2 is set to, for example, “255”. In the example of FIG. 4C, the number of black regions and white regions that are alternately repeated is four as in FIG. 4A, but may be five or more. The image data of the black and white pattern image OP2 may be stored in advance in the storage unit of the video reproduction device 1, or may be transmitted to the video reproduction device 1 together with a projection command for the black and white pattern image OP2. When receiving the projection command for the black and white pattern image OP <b> 2 from the projection control device 6, the video reproduction device 1 reproduces the black and white pattern image OP <b> 2 shown in FIG. 4C and outputs the image signal to the projector 2. The projector 2 projects the black-and-white pattern image OP2 as a projection image on the projection plane in the projection range according to the reproduced and output image signal. There is no person in the projection range at this time.

Next, the control unit 63 transmits an imaging command for the black and white pattern image OP2 to the camera 3 (step S35). When the camera 3 receives the imaging command for the black and white pattern image OP2 from the projection control device 6, the black and white pattern image OP2 within the imaging range that falls within the angle of view of the camera 3 is taken as a captured image P2 as shown in FIG. The captured image P <b> 2 is captured and transmitted to the projection control device 6. In the captured image P2 shown in FIG. 4D, as in FIG. 4B, the gray gradation decreases as the white region has a larger distance in the y-axis direction from the origin (0, 0). Next, the control unit 63 acquires (receives) the captured image P2 from the camera 3 and records it (step S36). Thereby, for example, the image data of the captured image P2 and the identification ID of the captured image P2 are associated with each other and stored in the storage unit 62. Similar to the case of capturing the monochrome pattern image described above, when the control unit 63 using the association table acquires the captured image P2 from the camera 3, the captured image P2 is obtained in accordance with the association table created in advance. The camera coordinates (x _c , y _c ) are converted into the projector coordinates (x _p , y _p ), and the captured image P2 with the converted coordinates is recorded (step S36).

  Next, the control unit 63 generates a composite image P3 (FIG. 4E) in which the white areas of the captured image P1 recorded in step S33 and the captured image P2 recorded in step S35 are combined. Recording is performed (step S37). For example, the control unit 63 has a gradation value (for example, 254) of a pixel (for example, a pixel at a certain coordinate (300, 200) in the white region) that constitutes the captured image P1 and a pixel (for example, for example) In the black area, a gradation value (for example, 252) corresponding to the difference (absolute value) from the gradation value (for example, 3) of the pixel (300, 200) is set as the gradation value of the pixel. . A composite image P3 is generated by performing such processing for all pixels.

Next, the control unit 63 creates a reflectance map from the composite image P3 generated in step S37 (step S38). The reflectance here means the reflectance on the projection surface of the light emitted from the projector 2. The region closer to the installation position of the projector 2 (subject region) is considered to have a higher reflectance of light from that region. Therefore, the intensity of light from the projector 2 is estimated by determining the reflectance. Is possible. The reflectance map is a map in which the reflectance is determined for each pixel. Each coordinate (x, y) in the reflectance map corresponds to each coordinate (x _p , y _p ) in the composite image P3. The higher the gradation value Gray (x _p , y _p ) of each pixel constituting the composite image P3 (closer to white), the higher the reflectance of the corresponding coordinate in the reflectance map. The reflectance is calculated by the following equation (1), for example. Note that (x, y) is a function indicating coordinates.

Reflectivity R (x, y) = (255−Gray (x _p , y _p )) / 255 (1)

  Next, the control unit 63 creates a linear interpolation LUT from the reflectance map created in step S38 (step S39), and returns to the process shown in FIG. For example, as shown in FIGS. 6A and 6B, the control unit 63 inverts each reflectance R (x, y) in the reflectance map and calculates the projection intensity L (x, y). The projection intensity means the intensity of light emitted from the projector 2. In the captured image P3, since the gradation value is constant regardless of the change of the coordinate x in the x-axis direction, the coordinate x is omitted in FIGS. 6 (A) and 6 (B). Then, as shown in FIGS. 6B and 6C, the control unit 63 creates a linear interpolation LUT by obtaining a regression line in the y-axis direction of the calculated projection intensity L (x, y). The linear interpolation LUT created in this way has a gray level change that is defined in a coordinate system in which the gray level is lower as the reflectivity is higher (that is, the gray level varies depending on the reflectivity). It is an example of a table, and a gradation value LUT (x, y) is defined as a gray gradation for each pixel. The gray gradation is darker, that is, a gradation closer to black as the gradation is lower. According to the linear interpolation LUT, the projection intensity of the light applied to the region (subject region) where the reflectance of the light emitted from the projector 2 is relatively high is lowered, while the reflectance of the light is relatively low. It is possible to increase the projection intensity of the light applied to the region (that is, to increase the light emitted from the projector 2). In this example, LUT (x, y) = LUT (y), and as shown in FIG. 6D, the x coordinate is not related to the gradation value of the linear interpolation LUT. The linear interpolation LUT created in this way is stored in the storage unit 62, for example. The linear interpolation LUT is configured so as to be defined with an achromatic gray gradation (achromatic color), but may be configured with a chromatic (for example, RGB) gradation.

  In step S4, the control unit 63 determines whether or not there is a normal mode setting instruction. For example, when an instruction signal indicating a normal mode setting instruction is input to the projection control device 6 by the user operating the operating device 4 and specifying the normal mode setting button 52 displayed on the instruction receiving screen, the control unit 63, it is determined that a normal mode setting instruction has been issued. When it is determined that the normal mode setting instruction has been given (step S4: YES), the control unit 63 sets the normal mode as the execution mode of the mask correction process (step S5). On the other hand, when it is determined that there is no normal mode setting instruction (step S4: NO), the control unit 63 proceeds to step S6.

  In step S6, the control unit 63 determines whether there is a whitening mode setting instruction. For example, when an instruction signal indicating a whitening mode setting instruction is input to the projection control device 6 by the user operating the operation device 4 and specifying the whitening mode setting button 53 displayed on the instruction reception screen, the control unit 63, it is determined that there has been a whitening mode setting instruction. When it is determined that the whitening mode setting instruction has been received (step S6: YES), the control unit 63 sets the whitening mode as the execution mode of the mask correction process (step S7). On the other hand, if it is determined that there is no whitening mode setting instruction (step S6: NO), the control unit 63 proceeds to step S8.

  In step S8, the control unit 63 determines whether there is an instruction to start projection and imaging. For example, when the user operates the operation device 4 and designates the projection / imaging start button 54 displayed on the instruction reception screen, an instruction signal indicating a projection / imaging start instruction is input to the projection control device 6. The control unit 63 determines that there is an instruction to start projection / imaging. When it is determined that there is an instruction to start projection and imaging (step S8: YES), the control unit 63 proceeds to step S9. On the other hand, when the control unit 63 determines that there is no projection and imaging start instruction (step S8: NO), the control unit 63 proceeds to step S10. In step S10, the control unit 63 determines whether or not there is an end instruction. When it is determined that there is no termination instruction (step S10: NO), the control unit 63 returns to step S2. On the other hand, when it is determined that the end instruction has been given (step S10: YES), the control unit 63 ends the process shown in FIG.

  In step S9, the control unit 63 executes projection and imaging processing. Here, an example of the projection and imaging process will be described with reference to FIG. In step S91 shown in FIG. 7, the control unit 63 executes a difference image generation process. In the difference image generation process shown in FIG. 8A, the control unit 63 transmits a projection command for the background image B to the video reproduction device 1 (step S911). The background image B is, for example, the above-described original image (for example, FIG. 9A). The image data of the original image may be stored in advance in the storage unit of the video reproduction device 1 or may be transmitted to the video reproduction device 1 together with the projection command for the background image B. When receiving the projection command for the background image B from the projection control device 6, the video reproduction device 1 reproduces the background image B and outputs the image signal to the projector 2. The projector 2 projects the background image B as a projection image on the projection plane in the projection range according to the reproduced and output image signal. A person is present in the projection range at this time as shown in FIG. In the example of FIG. 1B, a person is shown as an example of an object, but an animal other than a person, an article, or the like may be used.

Next, the control unit 63 transmits an imaging command for the background image B to the camera 3 (step S912). When the camera 3 receives the imaging command for the background image B from the projection control device 6, the camera 3 converts the background image B (projected image) within the imaging range within the angle of view of the camera 3 to the captured image C <b> 1 (for example, FIG. 9B). The captured image C1 is transmitted to the projection control device 6. Next, the control unit 63 acquires (receives) the captured image C1 from the camera 3 and records it (step S913). Thereby, for example, the image data of the captured image C1 and the identification ID of the captured image C1 are associated with each other and stored in the storage unit 62. Similar to the case of capturing the monochrome pattern image described above, when the control unit 63 using the association table acquires the captured image C1 from the camera 3, the captured image C1 is obtained in accordance with the association table created in advance. The camera coordinates (x _c , y _c ) are converted into projector coordinates (x _p , y _p ), and the captured image C1 with the converted coordinates is recorded (step S913).

  Next, the control unit 63 generates and records a difference image D representing the difference between the projected background image B and the captured image C1 recorded in step S913 (step S914), and returns to the processing shown in FIG. . Thereby, for example, the image data of the difference image D and the identification ID of the difference image D are associated with each other and stored in the storage unit 62. The background image B does not include a person area, but the captured image C1 includes a person area, so that the difference image D (for example, FIG. 9C) is emphasized. For example, the control unit 63 determines the gradation value (D (x, y) corresponding to the difference (absolute value) between the gradation value of the pixels constituting the background image B and the gradation value of the pixels constituting the captured image C1. ) = | B (x, y) −C1 (x, y) |) is set as the gradation value of the pixel. The difference image D is generated by performing such processing for all the pixels. The gradation value here is, for example, an RGB value representing each color of RGB with 256 gradations. The difference between the gradation values of each pixel can be obtained by calculating the Euclidean distance between the coordinates of the RGB values in the RGB color space. In this case, since the difference between the gradation values in the area other than the person area is 0, it is represented in black.

  In step S92 shown in FIG. 7, the control unit 63 determines whether or not the whitening mode is set. When it is determined that the whitening mode is not set (step S92: NO), the control unit 63 proceeds to step S93. On the other hand, when it determines with the whitening mode being set (step S92: YES), the control part 63 progresses to step S94. In step S93, the control unit 63 reads the linear interpolation LUT created by the linear interpolation LUT creation process from the storage unit 62, and proceeds to step S97.

  In step S94, the control unit 63 reads the linear interpolation LUT created by the linear interpolation LUT creation process from the storage unit 62, and proceeds to step S95. In step S95, the control unit 63 detects a face area from the person area included in the captured image C1 recorded in step S912. For detection of a face area, for example, a Viola-Jones detection algorithm, or an algorithm such as face part (eyes, nose, mouth) detection or skin color detection is used. Next, in the linear interpolation LUT read in step S94, the control unit 63 sets any coordinate in the face area detected in step S95 (that is, in the area of the captured image C1 corresponding to the face area). The gray tone is corrected to create a whitening mode LUT (step S96), and the process proceeds to step S97. For example, the control unit 63 determines that the coordinates corresponding to the face area out of all the coordinates indicated by the linear interpolation LUT are the gray scales of the center coordinates in the face area (or coordinates within a predetermined vicinity range from the center coordinates). Create a whitening mode LUT by setting the tone to a vertex with a high gradation (peak), and for the area corresponding to the peripheral coordinates of that vertex, the gray gradation is newly set to a gradation that decreases linearly or nonlinearly. To do. For example, as shown in FIGS. 10A and 10B, the non-linear change of the gray gradation is represented by a gray gradation (tone value LUT ( xf, yf)) is the above vertex, and for example, it is defined by calculating a curve connected using an n (n> 2) function by regression. According to such a whitening mode LUT, the projection intensity of the light applied to the detected face area is increased (that is, the light emitted from the projector 2 is made stronger and brighter), and gradually as it moves away from it. In addition, the light projection intensity can be lowered.

  In step S97, the control unit 63 executes a mask correction image generation process. In the mask correction image generation process shown in FIG. 8B, the control unit 63 specifies one coordinate (x, y) in the difference image D generated in step S914 (step S971). Next, the control unit 63 determines whether or not the gradation value D (x, y) of the coordinates (x, y) specified in step S971 is less than a threshold value (step S972). When the control unit 63 determines that the gradation value D (x, y) of the coordinate (x, y) is less than the threshold (step S972: YES), the coordinate (x in the background image B projected in step S911) , y) is set as the gradation value M (x, y) of the coordinates (x, y) in the newly generated mask correction image M (step S973), and step S975. Proceed to On the other hand, when it is determined that the gradation value D (x, y) of the coordinates (x, y) is not less than the threshold (step S972: NO), the control unit 63 performs a mask correction process (step S974). In this mask correction process, the controller 63 reads the gradation value LUT (x, y) of the linear interpolation LUT read in step S93 or the gradation value LUT (x, y) of the whitening mode LUT created in step S96. y) is set as the gradation value M (x, y) of the coordinates (x, y) in the newly generated mask correction image M. That is, in the mask correction process, the control unit 63 compares the reflectance of the light emitted from the projector 2 within the area of the person existing in the projection range of the background image B according to the linear interpolation LUT or the whitening mode LUT. The tone of the “region of the mask correction image M” corresponding to a region that is relatively high is set low (set to the tone value LUT (x, y)), and the “mask corresponding to the region having a relatively low reflectance is set. The gradation of “area of corrected image M” is set high (set to gradation value LUT (x, y)). Accordingly, the mask correction image M (for example, FIG. 9) in which the mask correction processing is performed on a partial region of the background image B (original image) (that is, a region where the gradation value D (x, y) is equal to or larger than the threshold value). (D)) can be generated.

  In step S975, the control unit 63 determines whether there is a coordinate (x, y) that has not yet been specified in the difference image D. When the control unit 63 determines that there is a coordinate (x, y) that has not yet been specified (step S975: YES), the control unit 63 returns to step S971, and coordinates (x, y) that have not yet been specified in the difference image D. And the same processing as described above is performed. As a result, the control unit 63 sets, in the mask correction image M, a gray gradation in which the gradation increases in order from a region having a high light reflectance to a region having a low light reflectance in the region of the person specified by the difference image D. be able to. In particular, when the whitening mode LUT is used, the control unit 63 uses, for example, the gray scale of the center coordinate in the area of the mask correction image M as the coordinate corresponding to the face area detected in step S95. The gray level is set so that the gray level decreases in order as a high vertex, and the area outside the face area (that is, around the face area) starts from the coordinates corresponding to the outer edge of the face area, and starts from the area with high light reflectance. It can be set to the mask correction image M that sets a gray gradation in which gradation gradually decreases toward a low region. At this time, the gradation of the point having the highest gradation in the face area is set to be higher than the gradation of the area corresponding to the outer edge of the face area. The gradation of the face area is not limited to the case where only one coordinate gradation is used as a high gradation vertex, but the gradation of at least a part of the area is set to a gradation higher than the outer edge of the face area. It may be set. On the other hand, when it determines with the control part 63 not having the coordinate (x, y) which has not been specified yet (step S975: NO), it returns to the process shown in FIG.

  In step S98 illustrated in FIG. 7, the control unit 63 transmits a projection command for the mask correction image M generated by the mask correction image generation process to the video reproduction device 1. That is, the control unit 63 outputs a projection command for projecting an image including a region in which gray gradation is set by the mask correction process as a projection image. The image data of the mask correction image M is transmitted to the video reproduction apparatus 1 together with, for example, a projection command for the mask correction image M. When receiving the projection command of the mask correction image M from the projection control device 6, the video reproduction device 1 reproduces the mask correction image M and outputs the image signal to the projector 2. The projector 2 projects the mask correction image M as a projection image on the projection plane in the projection range according to the reproduced and output image signal. That is, the mask correction image M is projected onto the projection plane and the person by the video reproduction device 1 and the projector 2 in accordance with the projection command from the projection control device 6. A person exists within the projection range at this time, as shown in FIG. 1B, and the gradation of the area of the mask correction image M corresponding to the area where the reflectance of light is high in the area of the person. Since it is set low, the projection intensity of the light irradiated onto this region can be lowered. On the other hand, since the gradation of the area of the mask correction image M corresponding to the area where the reflectance of light is low in this person area is set high, the projection intensity of the light applied to this area can be increased.

  Next, the control unit 63 transmits an imaging command for the mask correction image M to the camera 3 (step S99). When the camera 3 receives an imaging command for the mask-corrected image M from the projection control device 6, the camera 3 projects the person existing in the imaging range (projection range) within the angle of view of the camera 3 and the projection surface and the person from the projector 2. The mask-corrected image M (projected image) is captured as a captured image C2, and the captured captured image C2 is transmitted to the projection control device 6. Next, the control unit 63 acquires (receives) the captured image C2 from the camera 3 and records it (step S100). Thereby, for example, the image data of the captured image C2 and the identification ID of the captured image C2 are associated with each other and stored in the storage unit 62. Next, the control unit 63 associates the identification ID of the captured image C1 recorded in step S912, the identification ID of the difference image D recorded in step S914, and the identification ID of the captured image C2 recorded in step S100. The attached data is generated, the generated association data is stored in the storage unit 62 (step S101), and the process returns to the process shown in FIG. By recording the captured image C1, the captured image C2, and the difference image D in association with each other, these images can be used later for verification, image correction, and the like.

  As described above, according to the above-described embodiment, the projection control device 6 reflects the reflectance of light emitted from the projector 2 in the region of an object (for example, a person) that exists within the projection range of the background image B. The gradation of the area of the mask correction image M corresponding to a relatively high area is set low, and the gradation of the area of the mask correction image M corresponding to the area where the reflectance is relatively low is set high. Then, the mask correction image M is projected on the projector 2 as a projection image. Then, the projection control device 6 includes an object (for example, a person) existing within the projection range of the mask correction image M projected as a projection image by the projector 2, and a projection surface and an object (for example, a person) by the projector 2. Since the captured image is acquired and recorded from the camera 3 that has captured the mask correction image M projected on the captured image as a captured image, a smoother and natural image can be captured and recorded. . For example, when there are a region having a high reflectance and a region having a low reflectance from the projector 2 in the region of the person, it is possible to reduce the color difference that affects the photographing result of the camera 3 due to the light intensity. Further, as described above, by using the whitening mode LUT, it is possible to capture and record an image in which the face portion of the person looks better.

  In addition, according to the mask correction process of the present embodiment, it is not blurred as compared with a conventional simple mask process in which, for example, a person region is masked with a single black color. That is, in the conventional simple mask process, the person area is uniformly darkened. However, according to the mask correction process of the present embodiment, for example, the brightness of the person area can be made closer to uniform. Further, by masking a gray gradation in a partial area of the background image B (original image) (that is, an area where the gradation value D (x, y) is equal to or greater than the threshold value), depending on the arrangement of the projector 2, a person can be obtained. The shadow can be hidden even if a shadow is formed on a part of the body (such as the neck or the torso).

  In the above embodiment, the region of the person existing within the projection range is specified from the difference image D. However, for example, an infrared sensor or the like is separately provided, and the position information of the person is acquired by the infrared sensor or the like. The person area in the original image may be determined from the position information. Moreover, in the said embodiment, although comprised so that the reflectance of light might be specified from the synthesized image P3, the positional relationship between the projector 2 and a wall surface (projection surface) was memorize | stored beforehand, and a reflectance is calculated from the positional relationship. You may decide. In the above-described embodiment, the region where the reflectance of light is high and the region where the light reflectance is low is specified from the composite image P3 in the region of the person. However, the positional relationship between the projector 2 and the person is stored in advance. The region where the reflectance of light is high and the region where the light reflectance is low may be determined from the positional relationship.

DESCRIPTION OF SYMBOLS 1 Image | video reproduction apparatus 2 Projector 3 Camera 4 Operation apparatus 5 Display 6 Projection control apparatus

Claims (5)

First storage means for storing an image to be projected onto the projection plane by the projection device;
Detecting means for detecting an area of a person's face from an area of an object existing within the projection range of the image;
The gradation of the region of the image corresponding to the region where the reflectance of light emitted from the projection device is relatively high in the region of the object existing within the projection range of the image with respect to the image, Setting means for setting lower than the gradation of the region of the image corresponding to the region having a relatively low reflectance, and the region of the image corresponding to the region of the face detected by the detection unit is Setting means for setting a gray gradation having a gradation higher than that of the periphery of the region;
Control means for causing the projection apparatus to project the image including the area where the gradation is set by the setting means as a projection image;
Acquisition means for acquiring the captured image captured by the imaging apparatus using the target object existing within the projection range of the projected image and the projection image projected onto the projection plane and the target object from the projection apparatus as a captured image. When,
Second storage means for storing the captured image acquired by the acquisition means;
A projection control apparatus comprising:   The projection control apparatus according to claim 1, wherein the setting unit sets a gray gradation in which the gradation increases in order from the high reflectance region to the low reflectance region. Further comprising a creating means for creating a gradation change table in which a gray gradation having different gradations according to the reflectance is defined in a coordinate system in common with the image;
3. The projection according to claim 2, wherein the setting unit sets, in accordance with the gradation change table, a gray gradation in which gradation increases in order from the high reflectance region to the low reflectance region. Control device. Storing an image to be projected onto a projection plane by the projection device in a storage unit;
Detecting an area of a human face from an area of an object existing within a projection range of the image;
The gradation of the region of the image corresponding to the region where the reflectance of light emitted from the projection device is relatively high in the region of the object existing within the projection range of the image with respect to the image, A step of setting lower than a gradation of a region of the image corresponding to a region having a relatively low reflectance, and the region of the image corresponding to the detected region of the face is around the region Setting a higher gray level,
Projecting the image including the region where the gradation is set onto the projection device as a projection image;
Obtaining the captured image captured by the imaging device using the target object existing within the projection range of the projected image and the projected image projected onto the projection plane and the target object from the projection device as a captured image; ,
Storing the acquired captured image in the storage means;
A program that causes a computer to execute. A projection system comprising: a projection device that projects an image onto a projection plane; an imaging device that captures the projected image; and a projection control device that controls the projection,
The projection control device includes:
First storage means for storing an image to be projected onto a projection plane by the projection device;
Detecting means for detecting an area of a person's face from an area of an object existing within the projection range of the image;
The gradation of the region of the image corresponding to the region where the reflectance of light emitted from the projection device is relatively high in the region of the object existing within the projection range of the image with respect to the image, Setting means for setting lower than the gradation of the region of the image corresponding to the region having a relatively low reflectance, and the region of the image corresponding to the region of the face detected by the detection unit is Setting means for setting a gray gradation having a gradation higher than that of the periphery of the region;
Control means for outputting to the projection apparatus a projection command for projecting the image including the region in which the gradation is set by the setting means as a projection image;
With
The projection apparatus includes a projection unit that projects the projection image onto the projection plane and the object in response to the projection command from the projection control apparatus.
The imaging apparatus includes an imaging unit that captures, as a captured image, an object existing within a projection range of the projection image and the projection image projected onto the projection plane and the object from the projection apparatus,
The projection control device includes:
Acquiring means for acquiring the captured image captured by the imaging device;
Second storage means for storing the captured image acquired by the acquisition means;
A projection system comprising: