JP6140987B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to an image processing apparatus and an image processing method for adjusting luminance in a multi-projection system.

  The multi-projection system makes it possible to display an image with a resolution exceeding the performance of each projector by projecting an image by arranging a plurality of projectors. When projecting an image in a multi-projection system, projectors are generally installed such that projection areas of adjacent projectors partially overlap so that no gap is generated between the projection areas from the projectors. In a superimposition part (hereinafter referred to as a superimposition area) in the projection area, the luminance increases due to the overlap of light from each projector, so that the superimposition area and the other part (hereinafter referred to as a non-superimposition area) An unnatural luminance gap is generated at the boundary portion.

  As a technique for suppressing this gap, for example, there is an image luminance correction technique described in Patent Document 1. This document describes a method for calculating a luminance correction parameter for correcting the luminance value of each pixel of an image projected from each projector. Specifically, the luminance is constant in the region where the overlapping region and the non-overlapping region are combined (hereinafter referred to as the entire projection region), or the luminance at the boundary between the overlapping region and the non-overlapping region is smooth. It is described that correction is performed so as to change.

JP 2002-116500 A

  However, when the brightness correction parameter is calculated so that the brightness is constant in the entire projection area, the brightness value is made to be the same as the minimum brightness in the entire projection area before correction, so that the displayable brightness and contrast are reduced. Problems arise.

  In addition, the brightness at the projection center of each projector is generally higher than the brightness at the periphery. Therefore, even if correction is made so that the luminance change is smooth at the boundary between the superimposed region and the non-superimposed region, when viewed in the entire projection region, it is unnatural as if mountains with multiple luminance peaks are continuous. Brightness unevenness occurs.

An image processing apparatus according to the present invention includes an acquisition unit that acquires luminance characteristics of a plurality of projection units arranged so as to form a combined projection region so as to overlap at least a part of the projection region of each projection unit; Based on the acquired luminance characteristics of the plurality of projection means, the projected image data is corrected so that the luminance distribution between two points in one horizontal or vertical line of the combined projection area becomes a luminance distribution defined by a concave function. And a setting means for setting the luminance distribution within a range in which the luminance between two points in one horizontal or vertical line is a distribution defined by a concave function .

  According to the present invention, it is possible to perform a luminance correction process that suppresses a decrease in luminance and contrast while suppressing unnatural luminance unevenness in all projection areas.

FIG. 5 is a diagram illustrating an example of a relationship between a projector characteristic and a problem in the first embodiment. 1 is a diagram illustrating an overall configuration example of a multi-projection system in Embodiment 1. FIG. FIG. 3 is a block diagram illustrating an internal configuration example of an image processing unit according to the first embodiment. 3 is a flowchart illustrating an example of the flow of overall processing in the first embodiment. 6 is a flowchart illustrating an example of a process flow in a target luminance distribution generation unit according to the first embodiment. 6 is an explanatory diagram illustrating an example of target luminance distribution calculation for one horizontal line in Embodiment 1. FIG. It is explanatory drawing of an example of the target luminance distribution image generation in Example 1. FIG. FIG. 6 is an explanatory diagram of an example of target luminance distribution calculation when using three projectors in the first embodiment. 6 is an explanatory diagram illustrating an example of image division in Embodiment 1. FIG. 6 is a flowchart illustrating an example of a process flow in a luminance correction parameter calculation unit according to the first embodiment. FIG. 6 is a diagram for explaining an example of a luminance correction parameter in a superposed region in the first embodiment. FIG. 4 is a diagram illustrating an example of a brightness correction parameter calculated in the first embodiment. FIG. 10 is a block diagram illustrating an internal configuration example of an image processing unit according to a second embodiment. 10 is a flowchart illustrating an example of a flow of processing in the second embodiment. FIG. 10 is a diagram illustrating an example of a user interface for setting a target luminance distribution in the second embodiment. FIG. 10 is an explanatory diagram of an example of target luminance distribution calculation for one horizontal line in Example 3. It is explanatory drawing of an example which calculated the effect of Example 1 geometrically. It is explanatory drawing of an example of target luminance distribution calculation for 1 line of horizontal directions in Example 4. FIG.

[Example 1]
In general, the luminance distribution in the projection area of the projector is not uniform, and the luminance in the peripheral portion is lower than that in the central portion. Further, the maximum brightness that can be output varies depending on the lamp performance of each projector.

  Therefore, in this embodiment, by correcting the brightness at the center of the entire projection area and increasing the brightness at the periphery, the decrease in brightness and contrast in the entire projection area is minimized while suppressing unnatural brightness unevenness. Reduced brightness correction.

  Hereinafter, in this embodiment, for the sake of simplicity of explanation, a multi-projection system in which two projectors are arranged side by side in the horizontal direction will be described as an example. Two projectors are installed so that each projection area partially overlaps. At this time, it is assumed that the two projectors face the screen surface and are equidistant to the screen surface. Further, color differences and gamma characteristics between the projectors are corrected in advance, and the projector light can be shielded at an arbitrary timing.

  The luminance distribution before and after the luminance correction when two projectors are projected in the horizontal direction is as shown in FIG. The luminance distribution represents the luminance characteristics of each projector. FIG. 1A shows an image in which all the pixel values are R = G = B = 255 (that is, white) (hereinafter referred to as an all-white image) are simultaneously projected by two projectors, and the projected all-white image. The imaged image data obtained by imaging is represented. FIG. 1B shows a graph representing before and after correction of the luminance distribution along the dotted line portion in the captured image data. In FIG. 1B, the horizontal axis indicates the pixel position, and the vertical axis indicates the luminance value. The luminance value L can be calculated from the captured image data, for example, L = 0.22989 * R + 0.5866 * G + 0.1145 * B. Hereinafter, when the luminance distribution of the projection area is represented, the luminance distribution for one line in the projection area in the captured image data is represented, the horizontal axis represents the pixel position, and the vertical axis represents the luminance value. FIG. 1B shows that the maximum luminance value of projector A is higher than the maximum luminance value of projector B. As is clear from FIG. 1B, an unnatural luminance gap occurs before the correction in the overlapping region of the projector A and the projector B, but the unnatural luminance gap is eliminated after the correction. You can see that

  FIG. 2 is a diagram showing an example of the overall configuration of the multi-projection system in the present embodiment. An information processing apparatus 201 such as a PC for controlling a multi-projection system includes an input image data storage unit 202, an imaging control unit 203, a captured image data storage unit 204, an image processing unit 205, and an image projection control unit 206. The The information processing apparatus 201 can be an image processing apparatus that processes image data. In addition, an imaging device 207 such as a camera is connected to the information processing device 201 with a USB cable or the like, and projectors 208A and 208B are connected with an HDMI cable or the like. The input image data storage unit 202 stores image data to be displayed (hereinafter referred to as display image data) and calibration image data used for luminance correction. The calibration image data is, for example, image data indicating an all-white image in which all the pixel values are R = G = B = 255 (that is, white) as described above. The imaging control unit 203 controls the imaging device 207. The captured image data storage unit 204 stores captured image data captured by the imaging device 207. The image processing unit 205 corrects the luminance of the display image data based on the captured image data, and generates luminance corrected image data. The image projection control unit 206 controls the projectors 208 </ b> A and 208 </ b> B, and brightness correction indicated by the calibration image data indicated by the image data stored in the input image data storage unit 202 and the image data generated by the image processing unit 205. The projected image is projected. Note that all projection portions on the screen 209 when images are projected from the projectors onto the screen 209 are defined as projection regions 210, and a superimposed portion in the projection region is defined as a superimposition region 211. Thus, when the projection area 210 is formed so as to overlap at least a part of the projection area of each projector, it can be called a combined projection area.

  FIG. 3 is a block diagram showing an example of the internal configuration of the image processing unit 205 of this embodiment. The image processing unit 205 includes an image data input unit 301, a target luminance distribution generation unit 302, an image data division unit 303, a luminance correction parameter calculation unit 304, and a luminance correction processing unit 305. The image data input unit 301 acquires display image data F and calibration image data G stored in the input image data storage unit 202, and captured image data H stored in the captured image data storage unit 204. The target luminance distribution generation unit 302 generates target luminance distribution image data K that represents an ideal luminance distribution of the entire projection area based on the captured image data H. The image data dividing unit 303 divides the display image data F and the target luminance distribution image data K based on the captured image data H, and generates display divided image data f and target luminance distribution divided image data k. The luminance correction parameter calculation unit 304 calculates a luminance correction parameter W for correcting the luminance value of the display divided image data f based on the captured image data H and the target luminance distribution divided image data k. The luminance correction processing unit 305 corrects the luminance value of the display divided image data f based on the luminance correction parameter W, and generates luminance corrected image data f ′. Hereinafter, the width and height of the image indicated by the display image data F and the target luminance distribution image data K are u1 and v1, respectively, and the width and height of the image indicated by the captured image data G are u2 and v2, respectively. Further, the widths and heights of the screen resolution of the projector and the image indicated by the display divided image data f, the target luminance divided image data k, and the luminance corrected image data f ′ are u3 and v3, respectively. In order to simplify the description, it is assumed that the aspect ratios of all the projection areas in the display image data F and the captured image data G are the same.

  FIG. 4 is a flowchart showing the flow of processing in the multi-projection system of this embodiment. The overall processing flow in this embodiment will be described below using the flowchart of FIG. The processing shown in FIG. 4 is realized by the CPU executing a program stored in the RAM or ROM of the information processing apparatus 201.

  In step S401, the image projection control unit 206 controls each projector to project calibration image data G from each projector. And the imaging control part 203 controls the imaging device 207, and images the projection area | region 210 projected by each projector. In this step, two types of imaging are performed: individual imaging using the projectors 208A and 208B individually and overall imaging using both projectors simultaneously.

First, individual imaging will be described using the case of the projector 208A as an example. The individual imaging is executed in a state where the calibration image data G is projected from the projector 208A. A projection area 210 when projected onto the screen 209 from the projector 208A is a projection area 210A, and this is imaged by the imaging device 207 to obtain captured image data _HA . Captures an image Similarly, the projector 208B, obtaining captured image data H _B.

Next, whole imaging will be described. Whole imaging is performed in a state where the calibration image data G is simultaneously projected from each projector. The projection area 210 when projected onto the screen 209 from each projector is a projection area 210A + B, and the projection area 210A + B is imaged by the imaging device 207 to obtain captured image data H _{A + B.}

  The captured image data obtained by the individual imaging and the entire imaging are stored in the captured image data storage unit 204, respectively.

In step S402, the target luminance distribution generation unit 302 generates target luminance distribution image data K representing an ideal luminance distribution of the projection area 210A + B from the captured image data H _{A + B} obtained by the entire imaging. Details of generation of the target luminance distribution image data K will be described later.

In step S403, the image data dividing unit 303 generates display divided image data f _A and f _B projected from each projector from the display image data F. Further, the image data dividing unit 303 generates target luminance distribution divided image data k _A and k _B representing the ideal luminance distribution of the projection area of each projector from the target luminance distribution image data K. Details of the dividing methods of the display image data F and the target luminance distribution image data K will be described later.

In step S404, the luminance correction parameter calculator 304, the target luminance distribution divided image data k _A, with k _B, the display divided image data f _A, the luminance correction parameter for correcting the luminance value of f _B W _A , to calculate the W _B. Details of the method of calculating the brightness correction parameters W _A and W _B will be described later.

In step S405, the luminance correction parameter W _A, displayed on the basis of W _B divided image data f _A, the luminance value of f _B is corrected for each pixel, the luminance correction image data f _'A, f' to generate a _B. An example of a formula for calculating the luminance value in this embodiment is expressed by the following formula.

However, f'z (i, j) is a rounded integer value.

  Here, the pixel in the upper left corner of each image is the origin, the pixel position in the horizontal direction is i, and the pixel position in the vertical direction is j.

Finally, in step S406, the image projection control unit 206 controls each projector, projects the images indicated by the brightness correction image data f ′ _A and f ′ _B onto the screen 209 from each projector, and ends the process.

<Target luminance distribution image data generation>
Here, generation of target luminance distribution image data of the projection area 210A + B performed in step S402 will be described. The target luminance distribution image data K represents an ideal luminance distribution of the projection area 210A + B. The captured image and the display image are not necessarily the same size due to a difference in resolution. Therefore, before the target luminance distribution image data K is generated, pre-target luminance distribution image data K ′ indicating an image having the same size as the image indicated by the captured image data H _{A + B} is generated. The pre-target luminance distribution image data K ′ is obtained by obtaining the target luminance distribution for each horizontal line and calculating the target luminance distribution for all lines. Thereafter, target luminance distribution image data K indicating an image having the same size as the image indicated by the display image data F is generated based on the pre-target luminance distribution image data K ′.

  FIG. 5 is a flowchart showing the flow of processing in the target luminance distribution generation unit 302. This will be described below with reference to the flowchart of FIG.

First, in step S501, the target luminance distribution generation unit 302 generates pre-target luminance distribution image data K ′ indicating an image having the same size as the image indicated by the captured image data H _{A + B} , and sets the pixel value of each pixel, for example, It is initialized with 0 (that is, an all black image).

  Next, in step S502, the target luminance distribution generation unit 302 determines whether the target luminance distribution of all lines has been calculated for the pre-target luminance distribution image data K ′. If the calculation of the target luminance distribution has been completed for all lines, the process proceeds to step S508, and if not, the process proceeds to step S503.

  In step S503, the target luminance distribution generation unit 302 selects one horizontal line to be processed, and performs the processes of steps S504 to S507 on the one line.

In step S504, the target luminance distribution generation unit 302 acquires the pixel position and the luminance value of the representative point in the projection area 210H _{A + B} in the image indicated by the captured image data H _{A + B.} First, the target brightness distribution generating unit 302 includes a pixel P1, P2 corresponding to both ends of the projection area 210H _{A + B,} acquires the pixel P3 corresponding to the center of the projection area 210H _{A + B} as respective representative points. In addition, the target luminance distribution generation unit 302 acquires the pixel position i _Pm and the luminance value Li _Pm (m = 1, 2, 3) of the representative point, respectively. Hereinafter, in order to distinguish the projection area 210 on the screen 209 from the projection area in the image indicated by the captured image data H, the projection area in the image indicated by the captured image data H is designated as the projection area 210H _Z (Z = A, B, A + B, etc.).

In step S505, the target luminance distribution generation unit 302 calculates the target luminance distribution by obtaining a quadratic function that passes through the three representative points. In general, the following equation holds for a quadratic function y = ax ² + bx + c passing through three points (x ₁ , y ₁ ), (x ₂ , y ₂ ), (x ₃ , y ₃ ).

  This simultaneous equation can be expressed as a matrix as follows, and three coefficients a, b, and c can be obtained by using the Gauss-Jordan method.

In the present embodiment, (x ₁ , y ₁ ) = (i _P1 , L _P1 ), (x ₂ , y ₂ ) = (i _P2 , L _P2 ), (x ₃ , y ₃ ) = (i By obtaining the coefficients a, b and c as _P3 , L _P3 ), a quadratic curve connecting P1, P2 and P3 is obtained. This quadratic curve is set as a target luminance distribution for one line.

In step S506, the target luminance distribution generation unit 302 determines whether each projector can output according to the target luminance distribution. In the determination method, if the target luminance distribution falls within the luminance distribution of the projection area 210H _{A + B in} the captured image data H _{A + B} , output is possible, otherwise output is impossible. Specifically, the luminance value L _i of each pixel of the target luminance distribution calculated in step S505 is compared with the luminance value L ′ _i of the pixel at the same pixel position in the captured image data H _{A + B} , and all the pixels are compared. If L _i ≦ L ′ _i , output is possible, otherwise output is not possible. If output is possible, step S505 is completed and the process returns to step S502. Otherwise, the process proceeds to step S507.

  In step S507, the target luminance distribution generation unit 302 decreases the luminance value of the representative point P3 and returns to step S505. The reduction amount of the luminance value of the representative point P3 can be set to about 1/10 of the difference between the maximum luminance value and the minimum luminance value in one line, for example. However, since the number of processes in steps S505 to S507 increases when the reduction amount is small, the reduction amount can be set large when it is desired to shorten the processing time.

FIG. 6 shows an example of the calculated target luminance distribution for one line. A line 601 indicates the luminance distribution for one line in the projection area 210H _{A + B} in the captured image data H _{A + B} , that is, the luminance distribution before correction. A line 602 indicates a luminance distribution in a state where the representative points are simply connected. The luminance distribution of the line 602 includes target luminance that cannot be output by each projector. Therefore, in the present embodiment, the target luminance distribution for one line calculated by the processing of steps S505 to S507 like the line 603 is calculated by lowering the target luminance value of the representative point P3 and calculating the quadratic curve again. In this embodiment, the target luminance distribution is calculated together with a projector having a low outputable luminance among the plurality of projectors. As described above, as a characteristic of the projector, there is usually a tendency that the luminance value in the central portion is high and the luminance value decreases toward the peripheral portion. Therefore, in the present embodiment, as shown in FIG. 6, the luminance values of both ends of the projector A and the projector B are compared, and the luminance value L _P2 of the end i _P2 of the projector B having a low luminance value is compared. Set the luminance value of the end i _P1 . Then, the target luminance distribution is calculated. In this way, by calculating the target luminance distribution in conjunction with a projector with a low outputable luminance, it is possible to minimize the decrease in luminance and contrast even when the output characteristics of the other projector are extremely low. it can.

In the present embodiment, both ends and the central pixel of the projection area 210H A + _B of the image data in the H _{A + B} as the representative point, the target brightness distribution of one line by obtaining a quadratic curve connecting them An example of calculating the above has been described. However, the method of obtaining the curve is not limited to this as long as a luminance distribution is obtained in which the luminance of the central portion of the projection area 210A + B is high and the luminance of the peripheral portion is low. For example, the target luminance distribution may be calculated by increasing the representative points and obtaining a quartic function.

  Pre-target luminance distribution image data K ′ is generated by calculating the target luminance distribution in steps S503 to S507 for all lines. Thereafter, in step S508, the target luminance distribution generation unit 302 generates target luminance distribution image data K from the pre-target luminance distribution image data K ′. The reason for generating the target luminance distribution image data K from the pre-target luminance distribution image data K ′ is to eliminate the difference in image size due to the difference in resolution as described above.

Specifically, as shown in FIG. 7A, the pixel positions of the four corner pixels P ^{K ′} _{1 to} P ^{K ′} ₄ of the calculated target luminance distribution are acquired from the pre-target luminance distribution image data K ′. . Next, projective transformation is performed so that the pixels P ^{K ′} _{1 to} P ^{K ′} ₄ have the same size as the display image data F. In general, in the projective transformation, when the pixel position of the pre-conversion image is (x, y) and the pixel position of the post-conversion image is (x ′, y ′), the relationship between x, y, x ′, and y ′ is Can be represented.

In the above equation, a1 to a8 are projective transformation parameters between the pre-conversion image and the post-conversion image. Expression 4 is expanded, and pixel positions P ₁ (x ₁ , y ₁ ) to P ₄ (x ₄ , y ₄ ) in the pre-conversion image and pixel positions Q ₁ (x ′ ₁ , y ′) in the post-conversion image By substituting ₁ ) to Q ₄ (x ′ ₄ , y ′ ₄ ), the following simultaneous equations can be derived.

The projective transformation parameters a1 to a8 are calculated by solving the above equation. In this embodiment, the pixel positions P ₁ (x ₁ , y ₁ ) to P ₄ (x ₄ , y ₄ ) of the pre-conversion image are used as the pixel positions P ^{K ′} ₁ (x ₁ , x y ₁ ) to P ^{K ′} ₄ (x ₄ , y ₄ ). Also, the pixel positions Q ₁ (x ′ ₁ , y ′ ₁ ) to Q ₄ (x ′ ₄ , y ′ ₄ ) of the converted image are used as the pixel positions Q ^K ₁ (x ′ ₁ , y) of the target luminance distribution image data K. calculates a projective transformation parameter ^{_{as' 1) ~Q K 1 (x}} '4, y' 4). Then, by performing projective transformation based on the parameters, target luminance distribution image data K shown in FIG. 7B is obtained. The pixel positions Q ^K ₁ (x ′ ₁ , y ′ ₁ ) to Q ^K ₁ (x ′ ₄ , y ′ ₄ ) of the target luminance distribution image data K are obtained based on the display size of the display image data F. Can do.

  Through the processing described above, the target luminance distribution generation unit 302 generates target luminance distribution image data K indicating an ideal target luminance distribution of the projection area 210A + B.

In this embodiment, the case where two projectors are arranged in the horizontal direction has been described as an example. However, three or more projectors may be arranged in the horizontal direction. FIG. 8 shows an example in which three projectors are arranged in the horizontal direction and the processes of steps S501 to S507 are performed to calculate a target luminance distribution for a certain line. A line 801 indicates the luminance distribution of the projection area 210H _{A + B + C in} the captured image data H _{A + B + C} , that is, the luminance distribution before correction. A line 802 indicates a luminance distribution in a state where the representative points are simply connected. A line 803 indicates the target luminance distribution calculated by the processes in steps S505 to S507.

  Needless to say, the projectors may be arranged in the vertical direction. For example, by changing steps S502 to S507 to one line in the vertical direction instead of one line in the horizontal direction, it is possible to deal with the vertical direction. Further, when a plurality of projectors are arranged in the horizontal direction and the vertical direction, steps S502 to S507 are performed for each line in both the horizontal direction and the vertical direction, and the target luminance distribution image data K having the smaller luminance value as the target luminance. Can be generated.

<Image division>
Here, the image division processing performed by the image data dividing unit 303 in step S403 will be described with reference to FIG. Since each projector has a different degree of brightness correction, the target brightness distribution image data K and the display image data F are divided into image data of an image size output by each projector. By this processing, display divided image data f _A and f _B indicating an image projected by each projector and target luminance distribution divided image data k _A and k _B representing an ideal luminance distribution of the projection area of each projector are generated. To do. In generating the display divided image data f _A and f _B and the target luminance distribution divided image data k _A and k _B , projective transformation parameters are calculated. Hereinafter, a method for calculating the projective transformation parameter will be described.

FIG. 9A is a diagram illustrating an example of an image indicated by the captured image data H _{A + B} and the display image data F. First, the pixel positions of the pixels P ^H _{A1 to} P ^H _A4 and P ^H _{B1 to} P ^H _B4 at the four corners of the projection areas 210H _A and 210H _{B in} the captured image data H _{A + B} are acquired. The pixel positions of P ^H _A2 , P ^H _A3 , P ^H _B1 , and P ^H _B4 can be acquired based on, for example, a location where the luminance value changes rapidly as shown in FIG. Next, the pixels corresponding to the pixels P ^H _{A1 to} P ^H _A4 and P ^H _{B1 to} P ^H _B4 are acquired from the display image data F, and are set as P ^f _{A1 to} P ^f _A4 and P ^f _{B1 to} P ^f _B4 , respectively. . Specifically, P ^f _A1 , P ^f _A4 , P ^f _B2 , and P ^f _B3 are sequentially associated with pixels at the upper left corner, the lower left corner, the upper right corner, and the lower right corner of the display image data F, respectively. P ^f _A2, P ^f _B1 is the distance between the pixel ^{_{P H A1 P H B1, P}} H B1 P H distance between _A2, P ^H _A2 P ratio of the distance between the ^H _B2 are each pixel P ^f _A1 P ^f the distance between the _B1, the distance between the pixel P ^f _B1 P ^f _A2, is associated with a pixel that matches the ratio of the distance between P ^f _A2 P ^f _B2. Similarly, P ^f _A3 and P ^f _B4 are associated by using a ratio of distances between pixels. Finally, the projective transformation parameters (hereinafter referred to as parameters) described in step S507 from the pixels P ^H _{A1 to} P ^H _A4 in the captured image data H _{A + B} and the pixels P ^f _{A1 to} P ^f _A4 in the display divided image data f _A are used. (Denoted A). Similarly, projective transformation parameters (hereinafter referred to as parameter B) are obtained from the pixels at the four corners of the display divided image data f _B and P ^f _{B1 to} P ^f _B4 .

The image data dividing unit 303 generates the display divided image data f _A by performing the projective transformation from the display image data F and the parameter A, and performs the projective transformation from the display image data F and the parameter B to display the divided display image data f. Generate _B. FIG. 9B shows an example of an image indicated by the generated display divided image data f _A and f _B. Similarly, the image data dividing unit 303 performs target transformation using the target luminance distribution image data K and the parameter _A to generate target luminance distribution divided image data k _A. Then, the target luminance distribution divided image data k _B is generated by performing projective transformation using the target luminance distribution image data K and the parameter B.

Through the processing described above, the image data dividing unit 303 divides the display image data F and the target luminance distribution image data K, respectively, and displays the divided display image data f _A and f _B and the target luminance distribution divided image data k _A , Generate k _B.

<Brightness correction parameter calculation>
Here, the calculation of the brightness correction parameter performed by the brightness correction parameter calculation unit 304 in step S404 will be described. Brightness correction parameter calculation unit 304, a brightness correction parameter W _A, multiplied by calculating two parameters. In other words, the luminance correction parameter calculator 304, a luminance correction parameter W _A1 in the projection region 210H _A in the captured image data H _A, a luminance correction parameter W _A2 indicating a ratio of the luminance values projector 208A is outputted in overlapping region calculate. Brightness correction parameter calculation unit 304 obtains a luminance correction parameter W _A and calculated brightness correction parameter W _A1 and W _A2 by multiplying for each pixel. The brightness correction parameter W _B can be obtained by performing the same process on the captured image data H _B. These brightness correction parameters represent the ratio of adjusting the brightness of the image projected from each projector. By using these brightness correction parameters W _A and W _B , brightness correction of the display divided images f _A and f _B is performed.

  FIG. 10 is a flowchart showing the flow of processing in the brightness correction parameter calculation unit 304. Hereinafter, this process will be described with reference to the flowchart of FIG.

First, in step S1001, the luminance correction parameter calculation unit 304 corrects the luminance of the display divided image data f _A and f _B from the captured image data H _A and H _B and the target luminance distribution divided image data k _A and k _B. Parameters W _A1 and W _B1 are calculated. Specifically, the brightness correction parameter calculation unit 304 calculates the brightness correction parameter W _A1 using equation (6). In Expression (6), the luminance value Y _HA (i, j) of a predetermined pixel in the projection area 210H _A in the captured image data H _A is the luminance value Y _KA of the corresponding image of the target luminance distribution divided image data k _A. The brightness correction parameter _WA1 is calculated so as to be equal to (i, j). Incidentally, the corresponding pixel position of the pixel position and the target luminance distribution divided image data k _A of the projection area 210H _A in the captured image data H _A, for example by using the projective transformation parameters obtained during the above-described image divided corresponding Can be attached.

The brightness correction parameter W _B1 is calculated by performing the same process on the captured image H _B. The calculation formula of the brightness correction parameter W _B1 is expressed by the following formula.

Here, the luminance correction parameters W _{A1 and} W _B1 have values of 0 to _1, and the smaller the value, the smaller the luminance value of the display divided image data f _A and f _B. In the above process, the brightness correction parameter is calculated using the captured image data H _A and H _B , but the brightness correction parameter calculated using the captured image data H _{A + B} is used. Thus, the brightness correction parameter may be calculated.

Next, in step S1002, the luminance correction parameter calculation unit 304 binarizes the captured image data H _{A + B} and acquires the shape of the superimposed region 211. For example, first, the maximum luminance values Y _Ah and Y _Bh in the projection areas 210H _A and 210H _B in the captured image data H _A and H _B are obtained, and the larger numerical value is set as the threshold th. Next, binarization is performed by assigning 1 when the brightness value of the projection area 210H _{A + B in} the captured image data H _{A + B} is greater than or equal to th, and assigning 0 when the brightness value is less than th. The binary image data BI Generate _{A + B.} In subsequent step S1003, the overlapping region is identified by referring to each pixel of the binary image data BI _{A + B.} Note that the method of determining the threshold is not limited to this, and any value that can separate the overlapping region and the non-overlapping region may be used.

In step S1003, the luminance correction parameter calculation unit 304 calculates luminance correction parameters W _A2 and W _B2 that represent the ratio of the luminance value output by each projector in the overlapping region 211 having the shape acquired in step S1002. 11 (a) is an image showing the captured image data H _{A + B,} FIG. 11 (b) is an enlarged view of a superimposition region 211 of the image data in the H _{A + B.} If a certain pixel in the overlap region 211 is P (i, j), the luminance correction parameter _WA2 of this pixel can be expressed by the following equation.

Here, α represents the shortest horizontal distance from the point P to the peripheral edge of the projection area 210H _A , and β represents the shortest horizontal distance from the point P to the peripheral edge of the projection area 210H _B. . Similarly, the brightness correction parameter W _B2 in the superimposed region 211 in the captured image data H _B is calculated by the following equation.

As described above, the luminance correction parameters W _{A2 and} W _B2 in the superimposing region 211 represent the ratio of the luminance value output by each projector in each pixel in the superimposing region 211, and become 1 in total. FIG. 11C shows an example of the luminance correction parameters W _{A2 and} W _B2 for one horizontal line along the broken line portion of FIG.

In step S1004, the calculated final luminance correction parameter W _A by multiplying the luminance correction parameter W _A1 and luminance correction parameter W _A2 for each pixel. Similarly, to calculate the luminance correction parameter W _B by multiplying the luminance correction parameter W _B1 and a luminance correction parameter W _B2 for each pixel. Figure 12 horizontal one line of the luminance correction parameter along the broken line of (a) W _A, an example of W _B FIG. 12 (b), the shown (c). FIG. 12 (b) shows the luminance correction parameter W _A for performing brightness correction of the display splitting image data f _A, FIG. 12 (c) the luminance correction for performing brightness correction of the display splitting image data f _B It shows the parameters W _B.

By the above processing, the luminance correction parameter calculator 304, the captured image data _{H A, H B, H A} + B and a target brightness distribution divided image data k _A, from k _B, luminance correction for matching the target luminance distribution Parameters W _A and W _B are calculated. The luminance correction processing unit 305 obtains luminance corrected image data f ′ _A and f ′ _B by multiplying the display divided image data f _A and f _B for each pixel by the luminance correction parameters W _A and W _B. Then, the image projection control unit 206 projects the image indicated by the brightness correction image data f ′ _A and f ′ _B from each projector, thereby reducing the brightness and contrast of the projection area 210A + B while suppressing unnatural brightness unevenness. Luminance correction can be minimized.

  In the present embodiment, correction of luminance has been mainly described, but it goes without saying that the present embodiment may be applied to color components.

  The effect of this embodiment when geometrically calculated at the end of this embodiment will be described. Specifically, when an all white image is projected onto the screen and the brightness correction is performed so that the brightness of the entire projection area matches the minimum brightness in the same area, the brightness correction according to the present embodiment is performed. The contrast difference will be described.

First, the luminance distribution for one horizontal line in the projection areas of the projectors 208A and 208B is calculated from the captured image data H _A and H _B , respectively.

Next, the luminance distribution A calculated from the captured image data H _A is y = a ₁ x ² + b ₁ x + c ₁ , and the luminance distribution B calculated from the captured image data H _A is y = a ₂ x ² + b ₂ x + c ₂ . . Consider a case where the projection areas projected from the projectors 208A and 208B are adjusted so as to overlap each other by 25%, and the width of the luminance distribution A and the width of the luminance distribution B are 200 pixels. This state is shown in FIG. This figure is a sum of luminance values for one horizontal line in the captured image data H _A and H _B for each pixel. L _P1 indicates the luminance value at the pixel position i _{P1 at} the left end of the projection area 210A, and L _P2 indicates the luminance value at the pixel position i _{P2 at} the right end of the projection area 210B. L _P3 represents the luminance value at the pixel position i _P3 corresponding to the center of the projection area 210A + B.

Assuming that the peripheral light amount ratio between the projection area 210A and the projection area 210B is 70% and 73%, respectively, as an example of each coefficient, a ₁ = −0.0025, b ₁ = 0.55, c ₁ = 70, a ₂ = −0.003, b ₂ = 1.5, and c ₂ = −77.5.

In order to calculate the target luminance distribution from these two luminance distributions, the processes in steps S505 to S507 are repeated. As a result, the target luminance distribution indicated by the dotted line is calculated as y = −0.0008x ² + 0.3149x + 69.66.

  Finally, the difference between the luminance at the calculated vertex of the target luminance distribution and the lower luminance of the luminance at both ends is calculated. As a result of calculation using a tentative value, the minimum luminance value is 69.66 and the maximum luminance value is 100.1, so that an effect of a contrast increase of about 30% is expected.

  Note that, when the effect of the present embodiment is calculated geometrically, the captured image data H is used, but a data value measured using a colorimeter may be used.

[Example 2]
In the first embodiment, the calibration image G is projected from each projector on the screen 209, and the projection area 210 is captured using the imaging device 207 to obtain captured image data H. The method for determining the optimum target luminance distribution based on the obtained captured image data H and calculating the luminance correction parameter W has been described. In this embodiment, the user sets a target luminance distribution via a user interface (hereinafter referred to as UI). Also in the present embodiment, similarly to the first embodiment, the target luminance distribution is a distribution in which the luminance in the central portion is high in all the projection areas and the luminance is lowered in the peripheral portion. An example of calculating the brightness correction parameter W so as to match the distribution will be described.

  The configuration of the multi-projection system in the present embodiment is different only in the target luminance distribution generation unit 302 in the image processing unit 205 of the first embodiment. In this embodiment, two projectors will be described as an example. The projector performance and installation conditions are the same as those in the first embodiment.

  FIG. 13 is a block diagram illustrating the configuration of the image processing unit 205 of the present embodiment. In the target luminance distribution setting unit 1301, the user sets the target luminance distribution of the projection area 210A + B via the UI.

  FIG. 14 is a flowchart showing the flow of processing in the multi-projection system of the present embodiment. This will be described below with reference to the flowchart of FIG. Note that description of the same processing as in the first embodiment is omitted.

  In step S1401, the target luminance distribution setting unit 1301 sets the target luminance distribution according to an instruction from the user via the UI. FIG. 15 shows an example of a graphic user interface for setting the target luminance distribution. Projection region display unit 1501 for displaying projection region 210A + B, luminance distribution display unit 1502 for displaying a luminance distribution (solid line) and a target luminance distribution (broken line) along the dotted line in the projection region, and target luminance There is a target luminance distribution setting bar 1503 for setting the distribution. The user sets the target luminance distribution by sliding the target luminance distribution setting bar 1503 up and down. However, the brightness value of the pixels located at both ends of the projection area 210A + B is fixed, and the brightness value of the center pixel can be adjusted within a preset range, thereby increasing the brightness of the center part and the brightness of the peripheral part. The target luminance distribution can be set while maintaining the condition of lowering the value.

  When the user sets the target luminance distribution, the target luminance distribution setting unit 1301 determines whether each projector can output according to the target luminance distribution. The output determination method can be the same as in step S506. If output is possible, the process of step S508 is performed, and the target luminance distribution setting process is terminated. If the output is impossible, the user is notified that the output is impossible by a window message or the like, and the target brightness distribution is adjusted again by the target brightness distribution setting bar 1503. When the target luminance distribution setting process ends, steps S403 to S406 are performed.

  As described above, the user can set the target luminance distribution, and the luminance correction processing according to the target luminance distribution can be performed.

[Example 3]
In the present embodiment, an example in which the target luminance distribution generation process of the first embodiment is changed will be described. Except for the target luminance distribution generation processing, the same processing as in the first embodiment can be performed.

In the first embodiment, the representative points for one line are the pixels at both ends and the center in step S504. However, when the luminance distributions of the two projectors are greatly different as shown in FIG. 16, the target luminance distribution calculation method described in the first embodiment is the luminance of the projector that can output a high luminance among the two projectors. The property cannot be utilized. Therefore, first, the representative point for one line is set to only pixels at both ends, and these two representative points are connected to form a concave function (upward convex function) f (x). This function is lower than the graph of f (x) regardless of what two points are selected from the graph of f (x). Next, a luminance distribution g (x) for one line in the captured image H _{A + B} before correction is set. The final target luminance distribution f ′ (x) can be obtained by obtaining ΔE that minimizes ΔE = f (x) −g (x). However, it is assumed that the pixel value satisfies f (x)> g (x) in all pixels of one line. That is, the target luminance distribution is calculated so that the luminance distribution between two points in one horizontal or vertical line becomes a luminance distribution defined by the concave function.

  Also in the present embodiment, it is possible to perform a luminance correction process in which a decrease in luminance and contrast is minimized while suppressing unnatural luminance unevenness.

[Example 4]
In the fourth embodiment, an image in which all the pixel values are R = G = B = 0 (that is, black) (hereinafter, an image in which R = G = B = 0 is referred to as an all-black image) or pixel values of all the pixels are set. Processing when an image that is as close to 0 as possible is projected will be described. Due to the structure of the projector, when an all-black image or an image in which the pixel values of all the pixels are close to 0 is projected, there is a characteristic that it looks a little bright black because light is emitted from the projector. Hereinafter, this characteristic is expressed as black float. In the multi-projection system, the seam is conspicuous because black floating appears strongly in the overlap region. Therefore, in this embodiment, an example will be described in which, when an all-black image is displayed, the joint of the overlapping area generated due to black floating is inconspicuous.

  The outline of the flowchart of this embodiment is different from steps S401 and S402 described in the first embodiment, and steps S403 to S406 are the same. Hereinafter, the changes in steps S401 and S402 will be described.

  In step S401, that is, in the calibration image data projection / photographing process, the change point is that instead of projecting the all-white image as the calibration image data G, the all-black image is projected. That is, the luminance distribution of the combined projection area is based on the lowest luminance value that can be output by each projector.

  The change points in step S402, that is, the target luminance distribution image data generation process, are the output possibility determination of each projector, which is the process of step S506, and the luminance value change method of the representative point, which is the process of step S507.

In this embodiment, the determination method is different from that in the first embodiment in step S506. In the determination method in the present embodiment, output is possible when the target luminance distribution exceeds the luminance distribution of the projection area 210A + B in the captured image data H _{A + B} , and output is impossible otherwise. Specifically, the luminance value L _i of each pixel of the target luminance distribution calculated in step S505 is compared with the luminance value L ′ _i of the pixel at the same pixel position in the captured image data H _{A + B} , and all the pixels are compared. If L ′ _i ≦ L _i , output is possible, otherwise output is not possible. If output is possible, step S505 is completed and the process returns to step S502. Otherwise, the process proceeds to step S507.

  In this embodiment, in step S507, the target luminance distribution generation unit 302 increases the luminance value of the representative point P3, and returns to step S505. The increase value of the luminance value of the representative point P3 is the same as the increase value of the luminance value described in the first embodiment.

FIG. 18 shows an example of the calculated target luminance distribution for one line. A line 1801 indicates the luminance distribution for one line in the projection area 210A + B in the captured image data H _{A + B} , that is, the luminance distribution before correction. A line 1802 indicates the luminance distribution in a state where the representative points are simply connected. In the luminance distribution of the line 1802, the seam of the overlapping region generated by black floating becomes conspicuous. Therefore, in this embodiment, the target luminance value of the representative point P3 is increased and the quadratic curve is calculated again, and the target luminance distribution for one line calculated by the processing in steps S505 to S507 like the line 1803 is calculated.

  In the present embodiment, by making the luminance value of each pixel of the target luminance distribution higher than the luminance value of the pixel at the same pixel position of the captured image data, it is possible to make the black floating due to the characteristics of the projector projection inconspicuous.

  As described above, it is possible to perform a luminance correction process that suppresses unnatural brightness unevenness while minimizing black float and reduction in luminance and contrast.

<Other examples>
In the example described above, a multi-projection system in which a plurality of projectors and imaging devices are connected to the image processing apparatus has been described as an example. However, for example, a configuration equivalent to each embodiment may be adopted using a multimedia terminal in which an imaging unit and a projection unit are integrated in an image processing apparatus. When such a multimedia terminal has a plurality of projection means, the above-described embodiments can be realized by a single multimedia terminal. On the other hand, when such a multimedia terminal has only a single projection means, a system using a plurality of these multimedia terminals may be adopted. In this case, the same effects as those of the above-described embodiments can be obtained by exchanging the data between the multimedia terminals via the communication means.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (9)

Acquisition means for acquiring luminance characteristics of a plurality of projection means arranged to form a combined projection area so as to overlap at least a part of the projection area of each projection means;
Based on the acquired luminance characteristics of the plurality of projection means, the image data to be projected is adjusted so that the luminance distribution between two points in one horizontal or vertical line of the combined projection area becomes a luminance distribution defined by a concave function. Correction means for correcting ;
An image processing apparatus comprising: setting means for setting the luminance distribution within a range in which luminance between two points in one horizontal or vertical line is a distribution defined by a concave function . An image pickup means for picking up calibration image data projected from the plurality of projection means;
The image processing apparatus according to claim 1 , wherein the correction unit acquires a luminance characteristic of the projection unit by imaging with the imaging unit. The image processing apparatus according to claim 1 , wherein the correction unit determines the luminance distribution based on luminance values of pixels located at both ends and a center of the combined projection region. Wherein the correction means, according to claim 3, characterized in that the luminance values of the pixels located at the ends, determining the luminance distribution by applying the luminance value of the low projection means luminance characteristics among the plurality of projection means An image processing apparatus according to 1.   The image processing apparatus according to claim 1, wherein the luminance distribution of the combined projection area is based on a minimum luminance value that can be output by each projector. The image processing apparatus according to claim 5 , wherein the correction unit determines the luminance distribution so that the luminance distribution is higher than the lowest luminance value of a projection region corresponding to the distribution. A multi-projection system composed of a plurality of projectors arranged to form a combined projection region so as to overlap at least a part of the projection region of each projector,
Correction for correcting the projected image data based on the luminance characteristics of the plurality of projectors so that the luminance distribution between two points in one horizontal or vertical line of the combined projection area is a luminance distribution defined by a concave function Means,
A multi-projection system comprising: setting means for setting the luminance distribution within a range in which luminance between two points in one horizontal or vertical line is a distribution defined by a concave function . An acquisition step of acquiring luminance characteristics of a plurality of projection means arranged to form a combined projection area so as to overlap at least a part of the projection area of each projection means;
Based on the acquired luminance characteristics of the plurality of projection means, projection is performed so that the luminance distribution of the combined projection area is a luminance distribution defined by a concave function between two points in one horizontal or vertical line. A correction step for correcting the image data to be performed;
An image processing method comprising: a setting step of setting the luminance distribution within a range in which luminance between two points in one horizontal or vertical line is a distribution defined by a concave function . A program for causing a computer to execute the image processing method according to claim 8 .

Images