JP6872239B2 - Image processing equipment, image processing method and image processing program - Google Patents

Description

The present invention relates to an image processing apparatus, an image processing method, and an image processing program that project an image onto a projection object whose shape can be changed.

Generally, a projector that corrects image data by taking a picture of the image data projected by the projector onto a projection target with a camera, comparing the image data input to the projector with the image data taken by the camera. The Camera (ProCam) system is known. In the ProCam system, by associating the pixels of the image data input to the projector with the pixels of the image data taken by the camera, the color is corrected for each pixel and appropriate optical correction is performed. be able to.

Further, there is a method of using the ProCam system to perform optical correction so as to cope with the reflection characteristics of the projection surface and the dynamic change of the projection surface (see, for example, Patent Document 1). In the method described in Patent Document 1, the response function is obtained from the luminance value of each color component for each projected image, and the color correction is performed by using the light distribution map showing the rate of change of the luminance from the reference point. As a result, Patent Document 1 makes it possible to display an image without being affected by the pattern on the screen even when the image data is projected on a screen whose one surface is not white, such as a pattern on the screen.

Japanese Patent No. 5590668

However, in general, if the shape of the projection target changes, it may not be possible to make an appropriate correction. For example, when an image is projected onto an unfixed projection object such as a curtain, the projection object may shake and the shape of the projection object may change. When the shape of the projection target changes due to the shaking of the curtain, the distance to the image formation by the projection of the projector changes. The correspondence between the pixels of the image data input to the projector and the pixels of the image data taken by the camera is broken, and it is not possible to perform appropriate optical correction for each pixel, and the shape can be changed. There is a problem that an image cannot be projected properly on a suitable projection target.

Therefore, an object of the present invention is to provide an image processing apparatus, an image processing method, and an image processing program that appropriately project an image onto a projection object whose shape can be changed.

In order to solve the above problems, the first feature of the present invention relates to an image processing device that projects an image onto a projection target whose shape can be changed. The image processing apparatus according to the first feature of the present invention includes an area of image data input to a projector and an area of image data captured by a camera in a state where the image data is projected onto a projection target whose shape has not changed. The value of the image data input to the projector, the reflectance of the projection target whose shape can be changed, and the image data on the projection target for the region-to-region correspondence data associated with A storage device that stores response function data related to the response function that indicates the relationship between the projected state and the value of the image data captured by the camera, and the original image data is optically corrected using the response function, and the corrected image. The camera photographed an optical correction unit that generates data, a projection image output unit that inputs corrected image data to the projector, and a state in which the projector projects the corrected image data onto a projection target whose shape can be changed. The amount of deviation between the captured image acquisition unit that acquires the captured image data, the correspondence between the processing target regions in the inter-region correspondence data, and the correspondence between the corrected image data and the captured image data between the processing target regions. It includes a deviation amount determining unit that determines the reflectance of the processing target area, and a response function updating unit that updates the response function and the inter-region correspondence data according to the deviation amount determined by the deviation amount determining unit and the reflectance of the processing target area. Regarding the deviation amount, the deviation amount determination unit shifts the correspondence relationship between the deviation amount candidate generation unit that generates a plurality of deviation amount candidates and the processing target area in the inter-region correspondence data for each of the plurality of deviation amount candidates. A reflectance calculation unit that calculates the reflectance corresponding to each of a plurality of deviation amount candidates using a response function based on each of the correspondence relationships, and a reflectance corresponding to each of the plurality of deviation amount candidates. Is provided, and a deviation amount selection unit for selecting the deviation amount and the reflectance in the captured image data from a plurality of deviation amount candidates is provided.

A target image generation unit for generating a target target image data in a state of being photographed by a camera from the original image data is further provided, and a processing target area is determined by a value of a predetermined area of the photographed image data and inter-region correspondence data. It may be a predetermined region having a difference of a predetermined threshold or more by comparing with the value of the region of the target image data corresponding to the region.

The deviation amount candidate generation unit generates a plurality of deviation amount candidates on the epipolar line passing through the processing target area in the captured image data, and the reflectance calculation unit generates the deviation amount candidate for each of the plurality of deviation amount candidates on the epipolar line. The reflectance corresponding to each of the plurality of deviation amount candidates may be calculated based on each of the shifted correspondences.

The deviation amount selection unit selects the reflectance having the smallest reflectance dispersion in the neighboring region including the region corresponding to the processing target region from the reflectances corresponding to each of the plurality of deviation amount candidates, and the selected reflection. The deviation amount corresponding to the rate may be selected as an appropriate deviation amount.

The response function may be Equation 1.

When the processing of the captured image acquisition unit, the deviation amount determination unit, and the response function update unit is repeated, the reflectance calculation unit may calculate the reflectance by Equation 2.

When the original image data is one frame data of the video data and the deviation amount determining unit determines the sequential deviation amount for a plurality of frame data of the video data, the reflectance calculation unit calculates the reflectance by Equation 2. You may.

A second feature of the present invention relates to an image processing method for projecting an image onto a projection object whose shape can be changed. In the second sugar morning image processing method of the present invention, the camera captures an area of image data input to the projector and a state in which the image data is projected onto a projection target whose shape has not changed. The value and shape of the image data input to the projector can be changed for the step of storing the inter-regional correspondence data associated with the data area in the storage device and the area associated with the inter-regional correspondence data by the computer. The step of storing the response function data related to the response function indicating the relationship between the reflectance of the projection target and the value of the image data taken by the camera in the state where the image data is projected on the projection target in the storage device, and the computer The shape of the original image data can be changed between a step of optically correcting the original image data using a response function and generating the corrected image data, a step of the computer inputting the corrected image data to the projector, and a step of the computer inputting the corrected image data. The step of acquiring the captured image data taken by the camera from the state in which the projector projects the corrected image data on the projection target, the correspondence relationship between the processing target areas in the inter-region correspondence data, and the corrected image. The step of determining the deviation amount of the correspondence between the data and the captured image data and the reflectance of the processing target area, and the response function according to the deviation amount determined by the computer and the reflectance of the processing target area. A step of updating the inter-regional correspondence data is provided. The steps to determine are the step of generating a plurality of deviation amount candidates for the deviation amount, and the correspondence relationship in which the correspondence relationship of the processing target area in the inter-region correspondence data is shifted for each of the plurality of deviation amount candidates. Based on, the response function is used to compare the step of calculating the reflectance corresponding to each of the plurality of deviation amount candidates with the reflectance corresponding to each of the plurality of deviation amount candidates, and to compare the plurality of deviations. A step of selecting the amount of deviation and the reflectance in the captured image data from the quantity candidates is provided.

A third feature of the present invention relates to an image processing program for causing a computer to function as the image processing device according to the first feature of the present invention.

According to the present invention, it is possible to provide an image processing device, an image processing method, and an image processing program that appropriately project an image onto a projection target whose shape can be changed.

It is a figure explaining the hardware composition and the functional block of the image processing apparatus which concerns on embodiment of this invention. It is a figure explaining the image processing system which uses the image processing apparatus which concerns on embodiment of this invention. It is a figure explaining the image which concerns the processing of the image processing apparatus which concerns on embodiment of this invention. It is a flowchart explaining the processing of the image processing apparatus which concerns on embodiment of this invention. It is a flowchart explaining the optical correction process by the optical correction part which concerns on embodiment of this invention. It is a figure explaining the epipolar line in the epipolar geometry. It is a figure explaining the direction which the optical correction part which concerns on embodiment of this invention searches for a deviation. It is a figure explaining the parameter of the response function which concerns on embodiment of this invention. It is a figure explaining an example of the process which the optical correction part which concerns on embodiment of this invention selects the optimum deviation.

Next, embodiments of the present invention will be described with reference to the drawings. In the description of the drawings below, the same or similar parts are designated by the same or similar reference numerals.

(Overview)
The image processing apparatus 1 according to the embodiment of the present invention shown in FIG. 1 projects an image capable of suppressing the influence of the shape change on a projection target whose shape can be changed. The image processing device 1 optically corrects the original image data 22 in the projector 2 to generate the corrected image data 23, inputs the corrected image data 23 into the projector 2, and the projector 2 outputs the corrected image data 23. The captured image data 24 captured by the camera 3 in the projected state is acquired from the camera 3. The image processing device 1 estimates the deviation of the correspondence between pixels due to the shape change of the projection target, and considers the estimated deviation of the correspondence between pixels to suppress the influence of the shape change of the projection target on the corrected image data 23. The generated and corrected image data 23 is input to the projector 2. As a result, the image processing apparatus 1 according to the embodiment of the present invention can perform projection while suppressing the influence of the shape change of the projection target.

In the embodiment of the present invention, the case where the projector 2 applies optical correction to the still image and projects the image will be described, but the present invention is not limited to this. For example, when the projector 2 projects an image, the optical correction according to the embodiment of the present invention is applied to the frame data of the image, and the optical correction obtained in the preceding frame is applied to the subsequent frame. May be applied.

The image processing system 9 in which the image processing apparatus 1 according to the embodiment of the present invention is used will be described with reference to FIG. The image processing system 9 includes a projector 2, a camera 3, and a projection target 4 in addition to the image processing device 1 shown in FIG. The projector 2 projects the corrected image data 23 obtained by the optical correction onto the projection target 4. The camera 3 photographs the state of being imaged by the projection target 4 by the projection of the projector 2, and inputs the captured image data 24 obtained by the imaging to the image processing device 1. The image processing device 1 appropriately updates the response function and the pixel-to-pixel correspondence based on the input captured image data 24, and performs optical correction using the updated response function and the pixel-to-pixel correspondence to perform optical correction. The corrected image data 23 obtained as a result of the above is projected onto the projector 2.

In the embodiment of the present invention, the projection target 4 is a shape-changing object, for example, a curtain. Generally, the upper end of the curtain is suspended from the rail, and the lower end is released without being fixed at all. Therefore, the curtain may sway and change its shape under the influence of wind, contact, etc., and the change in shape is more noticeably observed downward.

For example, a case where the projector 2 projects the image data shown in FIG. 3A onto the curtain will be described. FIG. 3B shows an image taken by the camera 3 in a state of being projected onto a curtain whose shape has not changed, and the image data of FIG. 3A is appropriately projected.

FIG. 3C shows a state in which the curtain sways and is projected onto the projection target 4 in which the shape is changed on the left side, and is optically corrected by the image processing device 1 according to the embodiment of the present invention on the right side. Shows the projected state of the image. On the left side, a striped pattern is formed at the bottom, which matches the characteristics of the curtain, which tends to change shape toward the bottom. On the other hand, on the right side, the striped pattern that occurred on the left side is not observed, and it can be seen that the influence of the shape change of the projection target 4 is suppressed.

Further, FIG. 3D shows a state in which the image is projected onto the projection target 4 in which the shape is changed and a state in which the image corrected by the image processing device 1 is projected, as in FIG. 3C. Compared with FIG. 3C, the projection target 4 is different in that the shape is further changed. On the left side of FIG. 3D, a striped pattern is formed not only below but also as a whole. On the other hand, on the right side, as in FIG. 3C, the striped pattern generated on the left side is not observed, and it can be seen that the influence of the shape change of the projection target 4 is suppressed.

When the shape of the projection target 4 changes, the reason why the striped pattern is generally observed as shown on the left side of FIGS. 3 (c) and 3 (d) is the image input to the projector 2 shown in FIG. 3 (a). It is conceivable that the correspondence between the pixels of the above and the pixels of the image taken by the camera 3 shown in FIG. 3B is broken.

In general optical correction, since the coordinate system of the image input to the projector 2 and the image output by the camera 3 are different, the pixels of the image input to the projector 2 shown in FIG. 3A are used in advance. , The pixel-to-pixel correspondence, which is the correspondence with the pixels of the image taken by the camera 3 as shown in FIG. 3 (b), is performed. After that, the pixel to be corrected is specified with reference to the image taken by the camera 3 shown in FIG. 3 (b). Based on the pixel-to-pixel correspondence calculated in advance, the pixel to be corrected in the image input to the projector 2 shown in FIG. 3A is specified, and the specified pixel is corrected.

When the shape of the projection target 4 does not change, the positions of the projector 2 and the camera 3 do not change, so that the pixel-to-pixel correspondence calculated in advance is maintained. Therefore, in a state where the positional relationship between the projector 2, the camera 3, and the projection target 4 is determined, the pixel-to-pixel correspondence may be specified, and the optical correction may be performed using the specified pixel-to-pixel correspondence.

However, when the shape of the projection target 4 changes, the distance between the projector 2 and the image formation on the projection target 4 and the distance between the camera 3 and the image formation on the projection target 4 are changed. As a result, the color (reflectance) on the projection target 4 changes, and the change occurs from the pixel-to-pixel correspondence calculated in a state where the projection target 4 does not change its shape. Therefore, if the correction is performed using the pixel-to-pixel correspondence calculated in the state where the shape of the projection target 4 has not changed even though the shape of the projection target 4 has changed, a pixel different from the pixel to be corrected is obtained. It will be corrected.

As a result, for example, the predetermined pixel of the image input to the projector 2 is corrected so as to be bright, but the other pixels are corrected so as to be bright, and the predetermined pixel is not corrected appropriately. , It remains dark and the pixels that do not need to be brightened are corrected to be brighter. Similarly, although it was intended that the predetermined pixels of the image input to the projector 2 were corrected to be dark, the other pixels were corrected to be dark, and the predetermined pixels were not properly corrected and were bright. Pixels that do not need to be darkened are corrected to be darker. Therefore, as shown on the left side of FIGS. 3 (c) and 3 (d), light and dark are clearly formed and a striped pattern is observed.

Therefore, the image processing device 1 according to the embodiment of the present invention estimates the changed pixel-to-pixel correspondence even if the pixel-to-pixel correspondence is changed according to the shape change of the projection target 4, and the estimated pixel-to-pixel correspondence is estimated. The response function and the pixel-to-pixel correspondence are updated according to the above, and optical correction is performed using the updated response function and the pixel-to-pixel correspondence. As a result, as shown on the right side of FIGS. 3 (c) and 3 (d), the image processing apparatus 1 according to the embodiment of the present invention has the shape of the projection target 4 even when projected onto the projection target in which the shape has changed. The effect of change can be suppressed.

Generally, as the projection target 4 by the projector 2, a wall whose shape does not change, a stretched screen, or the like is used, but according to the image processing device 1 according to the embodiment of the present invention, a curtain or the like is used. Even if the shape changes, it can be the projection target 4. Therefore, according to the image processing apparatus 1 according to the embodiment of the present invention, the options of the projection target 4 can be expanded, and the projection by the projector 2 can be realized in various scenes.

(Outline of processing)
An outline of the processing of the image processing method according to the embodiment of the present invention will be described with reference to FIG.

First, in step S1, the image processing device 1 calculates in advance the data required for optical correction as preprocessing. When the installation of the projector 2, the camera 3 and the projection target 4 is completed, the positional relationship between the projector 2, the camera 3 and the projection target 4 is determined, and this positional relationship is not changed during the processing of the image processing method. Therefore, in step S1, the image processing device 1 generates inter-pixel correspondence data (inter-region correspondence data) 11, epipolar line data 12, response function data 13, and the like based on this positional relationship.

In step S1, appropriate test image data is projected to calculate each data. For example, a pattern image is projected to calculate the inter-pixel correspondence data 11. Further, in order to calculate the response function data 13, for example, the method described in Patent Document 1 is used.

The inter-pixel correspondence data 11 corresponds to an image data area (pixel) input to the projector 2 and an image data area in which the camera 3 captures a state in which the image data is projected onto a projection target whose shape has not changed. It is the data to be attached. In the preprocessing of step S1, the inter-pixel correspondence data 11 is generated by projecting the pattern image on the projector 2.

The epipolar line data 12 is the data of the epipolar line E based on the epipolar geometry. In the preprocessing, the epipolar line E of each pixel is calculated and stored as the epipolar line data 12.

The response function data 13 is data relating to a response function indicating the relationship between the value of the image data input to the projector 2, the reflectance of the projection target, and the value of the image data captured by the camera 3 for a predetermined pixel. .. In the preprocessing of step S1, for example, by the method described in Patent Document 1, an initial response function is generated in consideration of the pattern of the projection target 4 and the like. The response function data 13 and the pixel-to-pixel correspondence data 11 are appropriately updated in the subsequent processing according to the shape change of the projection target 4.

When the preprocessing is completed, in step S2, the image processing apparatus 1 generates the target image data 21 from the original image data 22 by using an ideal response function (not shown). In step S3, the image processing device 1 generates the corrected image data 23 from the original image data 22 by the response function. The ideal response function is generated within the light intensity of the projector 2, the outputable color range, and the like.

In step S4, the image processing device 1 inputs the corrected image data 23 generated in step S3 to the projector 2, and the projector 2 projects the corrected image data 23 onto the projection target 4. The camera 3 photographs the state in which the corrected image data 23 is projected onto the projection target 4, and in step S5, the image processing device 1 acquires the corrected image data 23 from the camera 3.

Next, in step S6, the image processing device 1 determines whether or not the difference in pixel values between the pixel to be processed of the captured image data 24 and the pixel to be processed of the target image data 21 is equal to or greater than a predetermined threshold value. To do. If the image processing device 1 determines in step S6 that the difference in pixel values is equal to or greater than a predetermined threshold value, the process proceeds to step S7. If the difference between the pixel values is equal to or greater than a predetermined threshold value, the image processing apparatus 1 changes the correspondence between pixels due to the shape change of the projection target 4 in this pixel. More specifically, the projection target 4 It is judged that the shape has changed at the corresponding position of. In step S7, the image processing device 1 estimates the changed pixel-to-pixel correspondence and reflectance for the pixel to be processed, and updates the response function and the pixel-to-pixel correspondence in step S8.

If the difference between the pixel values is within a predetermined threshold value, it is determined that the correspondence between pixels has not changed in this pixel, and more specifically, the shape has not changed at the corresponding position of the projection target 4. , Step S6 to step S8 are performed on the next pixel.

When the processing of steps S6 to S8 is completed for each pixel, the image processing device 1 determines whether or not the end condition is satisfied in step S9. The end condition may be, for example, the time or the number of times to display the image data, or is an end instruction input from an input device (not shown) or the like.

If it is determined that the end condition is not satisfied, the process returns to step S3, and the image processing device 1 generates the corrected image data 23 from the original image data 22 by the response function and the pixel-to-pixel correspondence. The response function and the pixel-to-pixel correspondence used in step S3 are different from the initial response function and the pixel-to-pixel correspondence generated in step S1 used in the processing in the first step S3, and the updated response function and the pixel-to-pixel correspondence in step S8. It may be pixel-to-pixel correspondence.

In step S4, the image processing device 1 inputs the corrected image data 23 generated in step S3 to the projector 2, and the projector 2 projects the corrected image data 23 onto the projection target 4. The camera 3 photographs the state in which the corrected image data 23 is projected onto the projection target 4, and in step S5, the image processing device 1 acquires the corrected image data 23 from the camera 3, and for each pixel, steps S6 to S6 to Step S8 is repeated.

In step S6 described above, that is, step S6 to be executed first, the image was projected using the initial response function generated for the projection target 4 whose shape has not changed without being affected by wind or the like. The captured image data 24 of the corrected image data 23 and the target image data 21 are compared.

On the other hand, in the process of step S6 after the response function and the pixel-to-pixel correspondence are updated in the pixel whose shape has changed in the projection target 4, the projection target 4 when the response function and the pixel-to-pixel correspondence are updated most recently. Is determined according to the degree of difference between the shape of the above and the shape of the projection target 4 at the time of processing this time. When the captured image data 24 corresponding to the corrected image data 23 projected using the most recently updated response function and the pixel-to-pixel correspondence is not different from the target image data 21, the response function and the pixel-to-pixel correspondence are updated. It is determined that there is no difference between the shape of the projection target 4 and the shape of the projection target 4 at the time of shooting the captured image data 24, and the correction is appropriately performed.

On the other hand, when the captured image data 24 corresponding to the corrected image data 23 projected using the recently updated response function and the pixel-to-pixel correspondence is different from the target image data 21, when the captured image data 24 is captured. It is considered that the shape of the projection target 4 has changed since the latest update of the response function and the inter-pixel correspondence, and the inter-pixel correspondence and the reflectance have shifted. In such a case, the image processing apparatus 1 further updates the most recently updated response function and the pixel-to-pixel correspondence in steps S7 and S8, and the response function and the pixel-to-pixel correspondence in which the original image data 22 is appropriately corrected. Search for correspondence.

In such an embodiment of the present invention, even when projecting onto the projection target 4 whose shape can be changed, the response function and the pixel-to-pixel correspondence are updated, and the updated response function and the pixel-to-pixel correspondence are used for correction. The projection of the image data is repeated. As a result, the response function and the inter-pixel correspondence can be sequentially updated following the shape change of the projection target 4, so that optical correction that is not easily affected by the shape change of the projection target 4 can be realized.

(Image processing device)
The image processing apparatus 1 according to the embodiment of the present invention will be described with reference to FIG.

The image processing device 1 is a general computer including a storage device 10, a processing device 30, and an input / output interface 50. The function shown in FIG. 1 is realized by executing an image processing program in which a general computer executes a predetermined process.

The storage device 10 is a ROM (Read Only Memory), a RAM (Random access memory), a hard disk, or the like, and stores various data such as input data, output data, and intermediate data for the processing device 30 to execute processing. .. The processing device 30 is a CPU (Central Processing Unit) that reads and writes data stored in the storage device 10 and inputs and outputs data to and from the input / output interface 50 to execute processing in the image processing device 1. To do. In particular, in the embodiment of the present invention, a GPU (Graphics Processing Unit) may be used as the processing device 30 for processing related to images. At this time, a three-dimensional graphics library such as OpenGL (registered trademark) may be used, and GLSL (OpenGL (registered trademark) Shading Language) or the like may be used to perform calculations on the shader.

The storage device 10 stores an image processing program that executes a predetermined process, and also stores inter-pixel correspondence data 11, epipolar line data 12, response function data 13, target image data 21, original image data 22, and corrected image data 23. , Captured image data 24, deviation candidate data 25, reflectance data 26, and selection deviation data 27 are stored.

The inter-pixel correspondence data 11, the epipolar line data 12, and the response function data 13 are data calculated in advance when the positions of the projector 2, the camera 3, and the projection target 4 are determined by the preprocessing unit 31, which will be described later.

The original image data 22 is image data to be processed by the image processing device 1. The corrected image data 23 obtained by optically correcting the original image data 22 is input to the projector 2 and projected. The target image data 21 and the corrected image data 23 are data generated from the original image data 22. The captured image data 24 is image data captured by the camera 3 in a state in which the projector 2 projects the corrected image data 23. The deviation candidate data 25, the reflectance data 26, and the selection deviation data 27 are data output by the deviation amount determining unit 36 described later.

The processing device 30 includes a preprocessing unit 31, a target image generation unit 32, an optical correction unit 33, a projected image output unit 34, a captured image acquisition unit 35, a deviation amount determination unit 36, and a response function update unit 37.

When the positions of the projector 2, the camera 3, and the projection target 4 are determined, the preprocessing unit 31 calculates in advance the inter-pixel correspondence data 11, the epipolar line data 12, and the response function data 13.

The inter-pixel correspondence data 11 is a pixel (area) of the image data input to the projector 2 and a pixel (area) of the image data in which the camera 3 captures a state in which the image data is projected on the projection target 4 whose shape has not changed. ) And the data. Since the coordinate systems of the images used by the projector 2 and the camera 3 are different, the preprocessing unit 31 generates the inter-pixel correspondence data 11 and enables the conversion of pixels in these different coordinate systems. Specifically, the pixel-to-pixel correspondence data 11 corresponds to the pixel identifier of the image data and the corresponding pixel identifier in the image data captured by the camera 3 for each pixel of the image data input to the projector 2. Attach. Here, the pixel identifier is, for example, the coordinates of the pixel in each image data. In the embodiment of the present invention, the pixel-to-pixel correspondence data 11 is sequentially updated as the shape of the projection target 4 changes.

For example, the preprocessing unit 31 inputs test image data to the projector 2 and compares the test image data with the image data taken by the camera 3 to generate inter-pixel correspondence data 11. The preprocessing unit 31 generates the inter-pixel correspondence data 11 in a state where the shape of the projection target 4 does not change.

The epipolar line data 12 is information on the epipolar line E calculated based on the epipolar geometry in the image data taken by the camera 3. The preprocessing unit 31 calculates the epipolar line E for each pixel associated with the pixel-to-pixel correspondence data 11, and generates epipolar line data 12 including information on the epipolar line E of each pixel.

The response function data 13 includes the value of the image data input to the projector 2, the reflectance of the projection target 4 whose shape can be changed, and the image data on the projection target 4 for the pixels associated with the pixel-to-pixel correspondence data 11. This is data related to a response function indicating the relationship between the projected state and the value of the image data captured by the camera 3. The value of the image data used in the response function data 13 may be a pixel value expressed in RGB or a brightness value calculated from the pixel value of RGB.

The response function data 13 is data of a response function for converting the original image data 22 into the corrected image data 23 to be input to the projector 2 in consideration of the characteristics of the projector 2 and the pattern of the projection target 4. The preprocessing unit 31 generates the response function data 13 by the method described in, for example, Patent Document 1. The response function generated by the preprocessing unit 31 is used to generate corrected image data 23 that suppresses the influence of the pattern, for example, when the projection target 4 has a pattern. In the embodiment of the present invention, the response function data 13 is sequentially updated as the shape of the projection target 4 changes.

The target image generation unit 32 generates target target image data 21 in a state of being photographed by the camera 3 from the original image data 22. The target image generation unit 32 generates the target image data 21 from the original image data 22 based on an ideal response function in consideration of the characteristics of the projector 2.

The target image data 21 is a comparison target when comparing whether or not the captured image data 24 captured by the camera 3 is appropriately optically corrected. The target image data 21 is generated in the same coordinate system as the image captured by the camera 3.

The optical correction unit 33 optically corrects the original image data 22 using the response function to generate the corrected image data 23. The optical correction unit 33 uses the response function specified by the response function data 13. Specifically, the optical correction unit 33 first generates the corrected image data 23 based on the initial response function generated by the preprocessing unit 31. After that, when the response function and the pixel-to-pixel correspondence are updated according to the shape change of the projection target 4, the corrected image data 23 is generated from the original image data 22 by using the updated response function and the pixel-to-pixel correspondence.

The projected image output unit 34 inputs the corrected image data 23 output by the optical correction unit 33 to the projector 2. The projector 2 projects the input corrected image data 23 onto the projection target 4.

The captured image acquisition unit 35 acquires the captured image data 24 captured by the camera 3 in a state in which the projector 2 projects the corrected image data 23 onto the projection target 4 whose shape can be changed. When the corrected image data 23 is input to the projector 2, the projector 2 projects the corrected image data 23 on the projection target 4. The camera 3 captures a state captured by the projection target 4 by the projector 2 and generates captured image data 24. The captured image acquisition unit 35 acquires the captured image data 24 from the camera 3 and captures the captured image data 24. Is stored in the storage device 10.

The deviation amount determining unit 36 determines the deviation amount and the processing target area of the correspondence relationship between the processing target pixels in the pixel-to-pixel correspondence data 11 and the correspondence relationship between the corrected image data 23 and the captured image data 24 between the processing target pixels. Determine the reflectance. The processing target pixels may be all the pixels of the corrected image data 23, or may be some pixels. When the processing target pixel is a part of the corrected image data 23, the part of the pixel is the value of the predetermined pixel of the captured image data 24 and the target image data 21 corresponding to the predetermined pixel in the inter-pixel correspondence data 11. It is a predetermined pixel having a difference of a predetermined threshold value or more by comparing with the value in the region of.

If the shape of the projection target 4 does not change after the response function and the pixel-to-pixel correspondence have been updated most recently, there is no significant difference between the target image data 21 and the captured image data 24 for the predetermined pixel. On the other hand, if the shape of the projection target 4 changes after the response function and the pixel-to-pixel correspondence have been updated most recently, the pixel-to-pixel correspondence collapses, so that there is a large difference between the target image data 21 and the captured image data 24 for a predetermined pixel. It will occur. In the embodiment of the present invention, the deviation amount determining unit 36 determines that the pixel having a large difference in pixel value between the target image data 21 and the captured image data 24 is affected by the change in the shape of the projection target 4. , The pixel is optically corrected by changing the correspondence between pixels. This makes it possible to reduce the processing load of the image processing device 1 and efficiently calculate an appropriate response function.

The response function updating unit 37 updates the correspondence between the response function and the pixels according to the deviation amount determined by the deviation amount determining unit 36 and the reflectance of the pixel to be processed. The response function update unit 37 updates the response function of the response function data 13 and the pixel-to-pixel correspondence of the pixel-to-pixel correspondence data 11 based on the deviation amount determined by the shift amount determination unit 36 and the reflectance of the pixel to be processed. After that, the optical correction unit 33 generates the corrected image data 23 from the original image data 22 according to the updated response function and the inter-pixel correspondence, and the corrected image data 23 is input to the projector 2.

(Deviation amount determination unit)
An outline of the processing of the deviation amount determining unit 36 according to the embodiment of the present invention will be described with reference to FIG.

First, in step S11, the shift amount determining unit 36 generates a plurality of shift amount candidates for shifting the pixels in the captured image data 24 in the epipolar line E direction with respect to the pixel-to-pixel correspondence in the pixel-to-pixel correspondence data 11. For example, with respect to the correspondence in the pixel-to-pixel correspondence data 11, a plurality of deviation amount candidates are generated, such as shifting 3 pixels to the left side of the epipolar line E, shifting 0 pixels (not shifting), shifting 3 pixels to the right side, and the like. To do.

In step S12, the deviation amount determining unit 36 calculates the reflectance for each of the plurality of deviation amount candidates generated in step S11 with reference to the response function data 13. When three deviation amount candidates are generated in step S11, the deviation amount determining unit 36 calculates three reflectances for the three deviation amount candidates.

In step S13, the deviation amount determining unit 36 selects a plausible reflectance deviation amount from the reflectances corresponding to the candidates for each deviation amount calculated in step S12. For example, the deviation amount determining unit 36 performs a process of calculating the reflectance for each pixel in the neighborhood region including the pixel to be processed for each deviation amount candidate, and for each deviation amount candidate, each pixel in the neighborhood region. The candidates for the deviation amount with the smallest reflectance dispersion are estimated as the plausible reflectance deviation amounts.

The deviation amount determining unit 36 outputs the deviation amount selected in step S13 as the deviation amount of the correspondence relationship between the corrected image data 23 and the captured image data 24 between the pixels to be processed. Further, among the reflectances calculated in step S12, the deviation amount determining unit 36 also outputs the reflectance corresponding to the determined deviation amount. The output deviation amount and reflectance are processed by the response function update unit 37.

As shown in FIG. 1, the deviation amount determining unit 36 includes a deviation amount candidate generation unit 41, a reflectance calculation unit 42, and a deviation amount selection unit 43. The processes of steps S11 to S13 of FIG. 5 correspond to the processes of the deviation amount candidate generation unit 41, the reflectance calculation unit 42, and the deviation amount selection unit 43, respectively.

Although the CPU may be used for the processing of the deviation amount determining unit 36, the GPU may be used. By performing parallel processing on the GPU, further speeding up of processing can be expected, so that processing can be performed more effectively than when a normal CPU is used.

The deviation amount candidate generation unit 41 generates a plurality of deviation amount candidates for the deviation amount of the correspondence relationship between the corrected image data 23 and the captured image data 24 between the processing target pixels. The deviation amount candidate generation unit 41 generates a plurality of deviation amount candidate candidates on the epipolar line E passing through the processing target pixel in the captured image data 24.

Due to the change in the shape of the projection target 4, the correspondence between the pixel to be processed in the target image data 21 and the pixel in the captured image data 24 is deviated, and the correspondence relationship in the pixel-to-pixel correspondence data 11 is changed. It is known that the pixels in the captured image data 24 corresponding to the pixels to be processed in the target image data 21 are displaced on the epipolar line E, but the amount of the deviation is unknown. Therefore, the deviation amount candidate generation unit 41 creates a plurality of pixel deviation amount candidates corresponding to the processing target pixel of the target image data 21 and the processing target pixel of the captured image data 24. The deviation amount candidate generation unit 41 generates deviation candidate data 25 including the generated deviation amount candidate information and stores it in the storage device 10.

Here, the epipolar line E will be described with reference to FIG. In epipolar geometry, a point p having image coordinates projected by the projector 2 is a point c projected onto the captured image data 24 captured by the camera 3, regardless of whether the image is formed by X, X', X'', or the like. It is known that c'and c'are observed on a specific line. Specifically, when the image formation at the point p of the image coordinates projected by the projector 2 is changed from X to X', the projection on the captured image data 24 taken by the camera 3 is changed from the point c to c'. Be changed. The specific line obtained by connecting the points c, c ′, and c ″ in this way is referred to as an epipolar line E. Since the colors observed at each coordinate on the epipolar line E correspond to different imaging positions, they are different from each other. The position of the projector 2 as seen from the camera 3 is referred to as an epipolar point. The epipolar point is the intersection of the straight line connecting the camera 3 and the projector 2 and the image plane.

The epipolar line E depends on the positional relationship between the projector 2 and the camera 3. Even when the shape of the projection target 4 changes, the processing target pixel shifts to any point on the epipolar line E of the captured image data 24. Therefore, in the embodiment of the present invention, as shown in FIG. 7, it is utilized that the position where the processing target pixel is displaced can be specified by searching on the epipolar line E of the captured image data 24.

Since the pixel to be processed shifts to any pixel on the epipolar line E due to the change in the shape of the projection target 4, the pixel in the target image data 21 and the pixel in the captured image data 24 are shifted. The pixel-to-pixel correspondence in the pixel-to-pixel correspondence data 11 cannot be used as it is. The pixels in the captured image data 24 are displaced on the epipolar line E, but the amount of the deviation is unknown. Therefore, the deviation amount candidate generation unit 41 generates some deviation amount candidates between the processing target pixel of the target image data 21 and the pixel of the captured image data 24 corresponding to the processing target pixel. The deviation amount candidate generation unit 41 stores the deviation candidate data 25 including the generated deviation amount candidate information in the storage device 10. The deviation amount determining unit 36 determines the candidate with the highest accuracy as an appropriate deviation amount in the process of assuming that each deviation amount candidate is correct.

The deviation amount candidate generation unit 41 may generate deviation amount candidates based on the deviation amount measured by the same optical system and the projection target 4, or randomly generate deviation amount candidates. Is also good. Since a shift occurs on the epipolar line E of the captured image data 24, the shift amount candidate generation unit 41 does not need to specify the shift amount in both the horizontal direction and the vertical direction. You only have to generate candidates.

In the embodiment of the present invention, the case where three candidates for the amount of deviation are generated will be described, but the present invention is not limited to this. For example, when the resources of the image processing device 1 have a margin and the transfer speed between the projector 2 and the camera 3 also has a margin, a larger number of candidates for the amount of deviation may be generated. Further, the number of deviation amount candidates generated by the deviation amount candidate generation unit 41 may be arbitrarily set in consideration of the resources of the image processing device 1 and the transfer speed with the projector 2 and the camera 3.

The reflectance calculation unit 42 uses a response function to shift the correspondence relationship of the pixels to be processed in the pixel-to-pixel correspondence data 11 for each of the plurality of deviation amount candidates, and uses a response function to perform a plurality of deviations. Calculate the reflectance corresponding to each of the quantity candidates. The reflectance calculation unit 42 stores the reflectance data 26 including the reflectance calculated for each of the plurality of deviation amount candidates in the storage device 10.

Here, the reflectance calculation unit 42 is a plurality of deviation amount candidates based on each of the correspondence relationships shifted for each of the plurality of deviation amount candidates on the epipolar line E passing through the processing target pixel of the captured image data 24. Calculate the reflectance corresponding to each of the above.

The response function according to the embodiment of the present invention is (Equation 1).

The relationship between P and C in (Equation 1) is shown in FIG. P relates to the input to the projector 2, and C relates to the output of the camera 3 when the camera 3 takes a picture of the state in which the input image to the projector 2 is projected onto the projection target 4.

Here, since the preprocessing unit 31 calculates the response function in advance, the projector output characteristic L is known. As the shape of the projection target 4 changes, the pixel to be processed shifts on the epipolar line E and the color changes. Therefore, in the response function of (Equation 1), the reflectances _{Ku and v} and the pixel correspondence u and v are used. Will be changed. Here, since the pixel correspondence u and v are assumed by the deviation amount candidates generated by the deviation amount candidate generation unit 41, the reflectance _{Ku and v for each deviation amount candidate can be obtained from the response function of (Equation 1).} Can be calculated.

Here, when the processing of the captured image acquisition unit 35, the deviation amount determination unit 36, and the response function update unit 37 is repeated, as shown in (Equation 2), immediately before the process is performed by the sequential least squares method (Recursive Least Square Algorithm: RLS). By using the calculation result used in the above process, it is possible to calculate the reflectance with high accuracy. (Equation 2) corrects the predetermined original image data 22 to generate the corrected image data 23, and inputs the corrected image data 23 to the projector 2 to generate a new corrected image data 23. It is suitable when the feedback is repeated.

Since the sequential least-squares method has a feature that the convergence to the target is faster than other adaptive algorithms, the reflectance can be appropriately calculated by the sequential least-squares method. Also in (Equation 2), sufficient if held only the R _t and S _t that used in the previous process, there is no need to hold all the past results. Therefore, the sequential least squares method can improve accuracy, suppress memory consumption, and realize high-speed processing.

Further, the image processing device 1 according to the embodiment of the present invention repeats feedback between the projector 2 and the camera 3, and repeats optical correction using the sequential least squares method. Therefore, even if there is a part where the effect is weak with one optical correction, the correction effect can be expected to be improved by repeating the optical correction.

For each of the plurality of deviation amount candidates generated by the deviation amount candidate generation unit 41, the reflectance calculation unit 42 reflects each pixel-to-pixel correspondence in which the pixel-to-pixel correspondence data 11 is changed according to each deviation amount candidate. Calculate the rate.

The deviation amount selection unit 43 selects the deviation amount in the captured image data 24 from the plurality of deviation amount candidates generated by the deviation amount candidate generation unit 41 based on the result of the reflectance calculation unit 42. The deviation amount selection unit 43 compares the reflectances corresponding to each of the plurality of deviation amount candidates, and selects the deviation amount in the captured image data 24 from the plurality of deviation amount candidates. The deviation amount selection unit 43 generates selection deviation data 27 including information on the selected deviation amount and stores it in the storage device 10. The deviation amount selected by the deviation amount selection unit 43 may differ from the actual deviation amount, but it is estimated that at least the deviation amount candidate generation unit 41 is closest to the actual deviation amount among the deviation amount candidates generated. Will be done.

There are several possible methods for the deviation amount selection unit 43 to select an appropriate deviation amount. For example, a method of selecting an appropriate amount of deviation that can output an image having a small variance of each reflectance of neighboring pixels in the captured image data 24 as an appropriate amount of deviation is appropriately compared with a corrected image generated from each amount of deviation. There is a method of selecting as the amount of deviation.

First, a method of utilizing the dispersion of reflectance when selecting a plausible amount from a plurality of candidates for a deviation amount will be described. The deviation amount selection unit 43 selects and selects from the reflectances corresponding to each of the plurality of deviation amount candidates the reflectance having the smallest reflectance dispersion in the vicinity region including the reflectance corresponding to the pixel to be processed. The amount of deviation corresponding to the reflectance is selected as an appropriate amount of deviation.

From the reflectance corresponding to each of the plurality of deviation amount candidates calculated by the reflectance calculation unit 42, the deviation amount selection unit 43 reflects with the smallest dispersion of reflectance in the vicinity region including the pixel corresponding to the pixel to be processed. Select the rate and select the amount of deviation corresponding to the selected reflectance as the appropriate amount of deviation. This utilizes the fact that when the correspondence between pixels is incorrect, the dispersion of reflectance (color) becomes large. For example, the dispersion of the color (reflectance) of each pixel in each image shown in FIGS. 9A to 9C is the smallest in FIG. 9A. Therefore, in the case of the example shown in FIG. 9, the deviation amount selection unit 43 can determine that the deviation of -4 pixels is caused by the shape change of the projection target 4.

When a method of selecting the amount of deviation using the reflectance is adopted, it is preferable that the reflectance calculation unit 42 calculates not only the reflectance of the pixel to be processed but also the reflectance of each pixel in the peripheral region. Alternatively, the reflectance calculation unit 42 may calculate only the reflectance of the processing target pixel, and the deviation amount selection unit 43 may calculate the reflectance of each pixel in the peripheral region of the processing target pixel.

Next, a method of comparing the corrected images generated from each deviation amount and selecting an appropriate deviation amount will be described. The deviation amount selection unit 43 generates a corrected image for each deviation amount candidate generated by the deviation amount candidate generation unit 41 by using each reflectance calculated by the reflectance calculation unit 42. The deviation amount selection unit 43 compares the corrected images corresponding to each deviation amount candidate. The deviation amount selection unit 43 selects, as an appropriate deviation amount, a deviation amount candidate corresponding to the corrected image in which the striped pattern observed in FIGS. 3C and 3D is not observed or the striped pattern is small. ..

The method using the dispersion of the reflectance has an advantage that the processing amount can be suppressed as compared with the method of comparing the corrected images generated from each deviation amount and selecting an appropriate deviation amount. In the method of comparing the corrected images generated from each deviation amount and selecting an appropriate deviation amount, it is necessary to create as many image data as the number of deviation candidates reflecting the calculated reflectance, and the image processing apparatus 1 May be burdensome. On the other hand, in the method using the reflectance dispersion, the calculated reflectance dispersion may be compared for each deviation candidate, and it is not necessary to generate a corrected image, so that the burden on the image processing device 1 is reduced. To.

In this way, by the processing of the shift amount determination unit 36, among the parameters of the response function in (Equation 1), the reflectance _{Ku, v} , the projector output characteristic L, and the pixel correspondence u, v are obtained, and the response function update unit 37 updates the response function data 13 and the inter-pixel correspondence data 11. Thus, optical correction unit 33, the camera observations C _u, the _v, by setting the value of the image as a target, the projector input value P _{u, v} can be calculated. Here, the value of the target image set in the camera observation values _{Cu and v} is the value of the target image data 21, and is represented by the coordinates of the camera coordinate system. The projector input values _{Pu and v} calculated in this way form the pixel values of the corrected image data 23 output by the optical correction unit 33.

As described above, the image processing apparatus 1 according to the embodiment of the present invention creates a candidate for the amount of deviation between pixels even when the shape of the projection target 4 changes and the correspondence between pixels shifts. The reflectance is calculated and evaluated for each candidate. As a result, the image processing device 1 can determine the amount of deviation from the candidates for the amount of deviation and output the corrected image data 23, so that it is possible to display an image in which the change in the shape of the projection target 4 is suppressed.

(First modification)
In the embodiment of the present invention, the image processing device 1 captures a region of image data input to the projector and a state in which the image data is projected onto a projection target whose shape has not changed, on a pixel-by-pixel basis. The case of associating data and correcting it has been described, but the present invention is not limited to this. For example, the image processing device 1 may make corrections by associating with a plurality of pixels composed of a plurality of pixels such as 2 pixels × 2 pixels and 8 pixels × 8 pixels.

As described above, in the embodiment of the present invention, the region serving as the processing unit by the image processing apparatus 1 may be composed of one pixel or a plurality of pixels.

When correcting in pixel units, a high-definition corrected image can be obtained, and when correcting in units of a plurality of pixels, speeding up of processing can be expected. The image processing device 1 includes the performance of various devices such as the image processing device 1, the projector 2 and the camera 3, the characteristics of the original image data 22, or the characteristics of the projection target 4, and the connection between the image processing device 1 and the projector 2 and the camera 3. The size of the area to be the processing unit may be set as appropriate according to the environment and the like.

(Second modification)
In the embodiment of the present invention, the case where the projector 2 projects a still image has been described, but it can also be applied to each frame data of an image. The original image data 22 according to the embodiment of the present invention corresponds to one frame data of the video data, and the optical correction unit 33 can be applied to a process of sequentially correcting a plurality of frame data of the video data. it can.

In this case, the optical correction unit 33 applies the deviation amount calculated for the preceding frame to the succeeding frame, and inputs the corrected succeeding frame to the projector 2. The optical correction unit 33, the _{R t} and _{S t} in Equation (2), by applying the processing result relating to the previous frame, and calculates the _{R t + 1} and _{S t + 1} for the subsequent frame.

As described above, the image processing device 1 according to the embodiment of the present invention projects not only when the shape of the projection target 4 changes while the still image is projected on the projection target 4, but also when the image is projected on the projection target 4. It can also be applied when the shape of the object 4 changes.

(Third variant)
In the embodiment of the present invention, the pixel-to-pixel correspondence data 11 is, for each pixel of the image data input to the projector 2, the pixel identifier of the image data and the corresponding pixel in the image data captured by the camera 3. The case of associating with an identifier has been described, but the present invention is not limited to this. For example, the inter-pixel correspondence data 11 may be data including the inter-pixel correspondence calculated by the preprocessing unit 31 and the history of the deviation amount determined by the deviation amount determination unit 36. Further, the inter-pixel correspondence data 11 may be data related to a conversion function representing the inter-pixel correspondence in which the deviation amount determined by the deviation amount determining unit 36 is reflected. The inter-pixel correspondence data 11 may be in any format as long as the latest inter-pixel correspondence can be grasped after the preprocessing unit 31 calculates the inter-pixel correspondence data 11.

(Other embodiments)
As described above, although described by embodiments of the present invention and first to third modifications thereof, the statements and drawings that form part of this disclosure should be understood to limit the invention. Absent. This disclosure reveals to those skilled in the art various alternative embodiments, examples and operational techniques.

For example, the image processing apparatus described in the embodiment of the present invention may be configured on one hardware as shown in FIG. 1, or may be configured on a plurality of hardware according to its function and the number of processes. May be done. Further, it may be realized on a computer that performs other processing.

It goes without saying that the present invention includes various embodiments not described here. Therefore, the technical scope of the present invention is defined only by the matters specifying the invention relating to the reasonable claims from the above description.

1 Image processing device 2 Projector 3 Camera 4 Projection target 9 Image processing system 10 Storage device 11 Interpixel correspondence data 12 Epipolar line data 13 Response function data 21 Target image data 22 Original image data 23 Corrected image data 24 Captured image data 25 Candidate data 26 Reflectivity data 27 Selection deviation data 30 Processing device 31 Preprocessing unit 32 Target image generation unit 33 Optical correction unit 34 Projection image output unit 35 Captured image acquisition unit 36 Deviation amount determination unit 37 Response function update unit 41 Deviation amount Candidate generation unit 42 Reflectivity calculation unit 43 Deviation amount selection unit

Claims (9)

An image processing device that projects an image onto a projection target whose shape can be changed.
Inter-regional correspondence data that associates the region of the image data input to the projector with the region of the image data captured by the camera when the image data is projected onto the projection target whose shape has not changed.
With respect to the regions associated with the inter-regional data, the value of the image data input to the projector, the reflectance of the projection target whose shape can be changed, and the state in which the image data is projected onto the projection target are described. A storage device that stores response function data related to a response function that indicates the relationship with the value of image data captured by the camera, and a storage device that stores the response function data.
An optical correction unit that optically corrects the original image data using the response function and generates corrected image data.
A projection image output unit that inputs the corrected image data to the projector, and
A captured image acquisition unit that acquires captured image data captured by the camera in a state in which the projector projects the corrected image data onto a projection target whose shape can be changed.
The amount of deviation of the correspondence relationship between the processing target areas in the inter-region correspondence data, the correspondence relationship between the corrected image data and the captured image data between the processing target areas, and the reflectance of the processing target area are determined. The deviation amount determination unit and
A response function update unit for updating the response function and inter-region correspondence data according to the deviation amount determined by the deviation amount determination unit and the reflectance of the processing target region is provided.
The deviation amount determining unit is
With respect to the deviation amount, a deviation amount candidate generation unit that generates a plurality of deviation amount candidates,
The correspondence relationship of the processing target area in the inter-region correspondence data is shifted for each of the plurality of deviation amount candidates, and the response function is used to shift the correspondence relationship of the plurality of deviation amount candidates. The reflectance calculation unit that calculates the reflectance corresponding to each of
It is characterized by including a deviation amount selection unit that compares the reflectances corresponding to each of the plurality of deviation amount candidates and selects the deviation amount and the reflectance in the captured image data from the plurality of deviation amount candidates. Image processing device. A target image generation unit for generating target target image data in a state captured by the camera from the original image data is further provided.
The processing target region compares the value of the predetermined region of the captured image data with the value of the region of the target image data corresponding to the predetermined region in the inter-region correspondence data, and the difference is equal to or greater than the predetermined threshold value. The image processing apparatus according to claim 1, wherein the predetermined area is provided. The shift amount candidate generation unit generates the plurality of shift amount candidates on the epipolar line passing through the processing target region in the captured image data.
The reflectance calculation unit calculates the reflectance corresponding to each of the plurality of deviation amount candidates on the epipolar line based on each of the corresponding relationships shifted for each of the plurality of deviation amount candidates. The image processing apparatus according to claim 1. The deviation amount selection unit selects and selects the reflectance having the smallest reflectance dispersion in the neighboring region including the region corresponding to the processing target region from the reflectance corresponding to each of the plurality of deviation amount candidates. The image processing apparatus according to claim 1, wherein a deviation amount corresponding to the reflected reflectance is selected as an appropriate deviation amount. The image processing apparatus according to claim 1, wherein the response function is the equation 1.

When the processing of the captured image acquisition unit, the deviation amount determination unit, and the response function update unit is repeated,
The image processing apparatus according to claim 5, wherein the reflectance calculation unit calculates the reflectance by the formula 2.

When the original image data is one frame data of the video data and the deviation amount determining unit determines the sequential deviation amount for a plurality of frame data of the video data.
The image processing apparatus according to claim 5, wherein the reflectance calculation unit calculates the reflectance by the formula 2.

An image processing method that projects an image onto a projection target whose shape can be changed.
The computer stores the inter-regional correspondence data in which the region of the image data input to the projector is associated with the region of the image data captured by the camera when the image data is projected onto the projection target whose shape has not changed. Steps to store in the device and
The computer projects the value of the image data input to the projector, the reflectance of the projection target whose shape can be changed, and the image data onto the projection target for the region associated with the inter-region correspondence data. A step of storing the response function data relating to the response function indicating the relationship between the state and the value of the image data taken by the camera in the storage device, and
A step in which the computer optically corrects the original image data using the response function to generate the corrected image data.
A step in which the computer inputs the corrected image data to the projector.
A step in which the computer acquires the captured image data captured by the camera in a state in which the projector projects the corrected image data onto a projection target whose shape can be changed.
The computer causes the deviation amount of the correspondence relationship between the processing target areas in the inter-region correspondence data and the correspondence relationship between the corrected image data and the captured image data between the processing target areas and the reflection of the processing target area. Steps to determine the rate and
The computer comprises a step of updating the response function and the inter-region correspondence data according to the determined deviation amount and the reflectance of the processing target region.
The step to determine is
With respect to the deviation amount, a step of generating a plurality of deviation amount candidates and
The correspondence relationship of the processing target area in the inter-region correspondence data is shifted for each of the plurality of deviation amount candidates, and the response function is used to shift the correspondence relationship of the plurality of deviation amount candidates. Steps to calculate the reflectance corresponding to each of
An image comprising a step of comparing the reflectances corresponding to each of the plurality of deviation amount candidates and selecting the deviation amount and the reflectance in the captured image data from the plurality of deviation amount candidates. Processing method. An image processing program for causing a computer to function as the image processing device according to any one of claims 1 to 7.

Images