JP7459644B2 - Projection system, projection method, program - Google Patents

Description

The present invention relates to a projection system, a projection method, and a program.

The use of multi-projection systems, which form one large screen by projecting from multiple projectors, is increasing. In a multi-projection system, multiple projectors each project an image to display one large image, so to avoid gaps between the images, installers position the projectors so that the images projected by adjacent projectors overlap in part.

The information processing device that transmits the images to the projector then performs image processing so that the overlapping parts of each image are not noticeable (see, for example, Patent Document 1). Patent Document 1 discloses a method for projecting a single high-resolution image by detecting the positions of feature points in a reference image captured by a digital camera, deforming the projected images of each projector based on the positions of four or more feature points of each projector that have been detected, detecting overlapping areas and performing blending processing, and arranging the deformed and blended images from multiple projectors on a screen while leaving overlapping areas between them.

However, the conventional technology has a problem in that it is difficult to suppress brightness while suppressing blackout in the overlapping area. That is, since the overlapping area where multiple images overlap has a higher brightness than the non-overlapping area, processing is performed to reduce the brightness by brightness adjustment (blending). Brightness adjustment is effective for relatively bright images, but for dark images, simply adjusting the brightness tends to cause blackouts. Therefore, the brightness cannot be lowered below a certain level. As a result, the overlapping area may become unnaturally brighter than the non-overlapping area.

In view of the above problems, the present invention aims to provide a projection system that can suppress brightness while suppressing crushed black in overlapping areas.

In view of the above problems, the present invention provides a projection system in which an information processing device transmits image data to a plurality of projection devices, and the plurality of projection devices each project an image onto a projection body, the information processing device having an overlapping area detection unit that detects an overlapping area of adjacent images from a captured image of the images projected onto the projection body, a gradation modification unit that changes the gradation of the image of the overlapping area detected by the overlapping area detection unit to a full range of 0 to 255 and changes the gradation of the image of the non-overlapping area to a limited range of 16 to 235 , a brightness adjustment unit that reduces the luminance of the overlapping area of the image whose gradation has been modified by the gradation modification unit, and a projection image output unit that transmits to the plurality of projection devices the image data in which the luminance of the overlapping area has been reduced by the luminance adjustment unit, and the plurality of projection devices receive the image data transmitted by the projection image output unit and each project an image.

It is possible to provide a projection system that can suppress brightness while suppressing blackout in overlapping areas.

FIG. 2 is a diagram illustrating an example of image processing performed by an information processing device. 1 is a schematic diagram showing the overall configuration of a projection system according to this embodiment. 1 is an example of a hardware configuration diagram of a projector. 1 is an example of a hardware configuration diagram of an information processing device. FIG. 1 is a functional block diagram of a projection system according to an embodiment of the present invention. 1A to 1C are diagrams illustrating two types of calibration images used in the projection system according to this embodiment. 1A to 1C are diagrams illustrating calibration projection scenes of a first aspect, which are sequentially selected by a calibration scene selection unit according to this embodiment, and a method for capturing the same. FIG. 7 is a diagram illustrating a second aspect of calibration projection scenes and an imaging method thereof, which are sequentially selected by the calibration scene selection unit according to the present embodiment. FIG. 11 is a flowchart illustrating an example of an overall process flow of a calculation process of various correction coefficients and a correction process based on the correction coefficients according to the present embodiment. 11 is an example of a flowchart illustrating a calculation process of a geometric correction coefficient executed by a correction coefficient calculation unit according to the present embodiment. FIG. 11A and 11B are diagrams illustrating three calibration captured images prepared by capturing each calibration projection scene in the calibration projection scene of the first aspect, and projective transformation coefficients calculated between these captured images. 13A and 13B are diagrams illustrating two calibration-use captured images prepared by capturing the respective calibration-use projection scenes in the calibration projection scenes of the second aspect, and projective transformation coefficients calculated between these captured images. FIG. 3 is a diagram schematically showing a set of grid point coordinates of each projector integrated on a common coordinate system. FIG. 7 is a diagram illustrating a method of calculating the outer circumferential coordinates of a projectable area by linear extrapolation using each integrated grid point coordinate. 11A and 11B are diagrams illustrating projectable areas of three projectors, corrected projection target areas, and projected content images on a common coordinate system. FIG. 6 is a diagram illustrating the correspondence between each coordinate on the projector memory and the coordinate on the same-size content image corresponding to the position on the projected content image. FIG. 11 is an example of a flowchart illustrating a blending coefficient calculation process executed by a correction coefficient calculation unit according to the present embodiment. 11 is a diagram illustrating an example of association of blending coefficients with each coordinate on a projector memory; FIG. 1 is an example of a graph showing input/output characteristics of a projector. It is a figure which illustrates the data structure of (A) a geometric correction coefficient and (B) a blending coefficient. FIG. 3 is a diagram illustrating gradation correction in the projection system. FIG. 3 is a diagram illustrating gradation correction from full range to limited range. FIG. 3 is a diagram illustrating gradation correction from limited range to full range. FIG. 6 is a diagram illustrating a correction process based on a correction coefficient, which is executed by a correction processing unit.

Below, an information processing device and a projection method performed by the information processing device will be described as an example of a form for implementing the present invention.

<Processing Overview>
First, an overview of the processing performed by the information processing device will be described with reference to Fig. 1. Fig. 1 is a diagram showing a schematic diagram of image processing performed by the information processing device. In Fig. 1, the information processing device is connected to three projectors (examples of projection devices) 150a to 150c.

Information processing device 110 transmits three images to each projector. The three images may be prepared in advance so as to have overlapping areas, or the information processing device 110 may divide one image into three. At the time of division, the information processing device 110 divides each divided image by overlapping the edges thereof by a predetermined width. The image divided and projected in this way is called a projected image. As described later, the three projectors 150a to 150c individually project the projection images, and one imaging device captures the three projection images. The information processing device 110 detects an overlapping area from the captured image. Therefore, the overlapping area of each projection image becomes clear.

One of the features of the information processing device 110 of this embodiment is that the gradation of the overlapping area of the projected images is wider than that of the non-overlapping area. For example, the RGB range of the non-overlapping area of the projected image is a limited range (eg 16 to 235), and the overlapping area is a full range (0 to 255).

Each of the projectors 150a to 150c projects the full RGB range. Projector 150a usually has an option for the user to set whether the input signal is limited range or full range. In this embodiment, it is set to full range.

Through this image processing of overlapping areas, the black level of non-overlapping areas converted to limited range is slightly raised (at least 16, so it becomes brighter), and overlapping areas that are full range are projected darker than non-overlapping areas. (The minimum value can be 0).

Note that since the information processing device 110 determines the values of the RGB range so that the brightness of the overlapping area is equal to the brightness of the non-overlapping area, the overlapping area is prevented from becoming brighter due to the overlapping of images.

Therefore, the information processing device 110 can display the overlapping areas darker. Because the overlapping areas are not corrected to be darker, they are less likely to be crushed black.

<Terminology>
An overlapping region is a region where multiple images overlap. It is preferable to detect only overlapping regions, but non-overlapping regions may also be included. In other words, it is sufficient if there is at least a partial overlapping region.

The gradation of an image is a number that indicates how many levels of color density and brightness can be expressed. Each color of RGB is expressed by how many bits. In this embodiment, the gradation of the original image is described as having 256 gradations, but it may be less than this or more than this.
<Overall composition>
Fig. 2 is a schematic diagram showing the overall configuration of the projection system 100 according to this embodiment. The projection system 100 shown in Fig. 2 includes an information processing device 110 that performs overall control of the system, a plurality of projectors 150, and a camera 160. In the embodiment to be described, the projection system 100 is configured to support so-called large-screen multi-projection, in which projected images from three projectors 150a to 150c are synthesized on a projection surface and an image is projected onto a larger area than that of a single projector.

Information processing device 110 is typically configured as a general-purpose computer such as a personal computer or a workstation. Note that the information processing device 110 is not limited to a general-purpose computer, and may be implemented as a dedicated controller or incorporated into any projector 150.

The projectors 150 are projection devices that employ a liquid crystal method, a CRT (Cathode Ray Tube) method, a DLP (Digital Light Processing) method, an LCOS (Liquid Crystal On Silicon) method, or the like.

Camera 160 is an imaging device that includes an image sensor such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD), and an imaging optical system such as a lens for forming an image on the light receiving area of the image sensor. Camera 160 may be configured as a dedicated device such as a web camera, digital still camera, or digital video camera, or may be configured as a device incorporated into a general-purpose device such as a smartphone terminal or tablet terminal.

The projection system 100 is provided with a screen 102 that provides a projection surface. The projectors 150 are provided so that they project onto the screen 102 while shifting the projection center position between the projectors. The information processing device 110 generates a plurality of projection images to be projected by the plurality of projectors 150a to 150c, and outputs each projection image to each of the corresponding projectors 150. The projectors 150 each project the projection image input from the information processing device 110 to the projector 150 onto the screen 102, which is a projection body. As shown in FIG. 2, a plurality of projection images 104a to 104c are projected onto the screen 102 from the plurality of projectors 150a to 150c. The plurality of projection images 104a to 104c from the plurality of projectors 150a to 150c are superimposed on the projection surface and synthesized into a single projection image 106.

During the projection mode, the projection system 100 projects a single projection image 106 using the multiple projectors 150a to 150c as described above, but a calibration process is usually performed before the projection mode described above. The camera 160 shown in FIG. 2 is used during this calibration process. During the calibration mode, the information processing device 110 outputs a calibration image to each of the multiple projectors 150, and causes them to project the calibration projection image onto the screen 102. Then, the camera viewpoint and field of view are set so that the projection image 104 on the screen 102 projected by a specific projector 150 falls within the angle of view of the camera 160, and the camera 160 captures images multiple times for calibration.

The captured image captured by the camera 160 (hereinafter, the captured image in which the projection image for calibration is reflected is referred to as the captured image for calibration) is connected to wireless LAN (Local Area Network), Bluetooth (registered trademark), wireless The information is transmitted to the information processing device 110 via a wireless connection such as a USB (Universal Serial Bus) or a wired connection such as a wired USB or wired LAN. Alternatively, the calibration image captured by the camera 160 is read by the information processing device 110 via a removable medium such as an SD card (registered trademark) or a compact flash (registered trademark).

The information processing device 110 uses the plurality of inputted calibration images to perform alignment, scale adjustment, distortion correction, and brightness adjustment of overlapping regions of each of the projection images of the plurality of projectors 150a to 150c. Calculate various correction coefficients. During the projection mode, the information processing device 110 generates corrected projection images to be projected by each of the projectors 150a to 150c based on the various calculated correction coefficients.

<Hardware configuration>
<<Projector>>
FIG. 3 is a hardware configuration diagram of the projector. As shown in FIG. 3, the projector 150 includes a CPU (Central Processing Unit) 801, a ROM (Read Only Memory) 802, a RAM (Random Access Memory) 803, a media I/F (Interface) 807, an operation unit 808, and a power switch. 809, bus line 810, network I/F 811, LED (Light Emitting Diode) drive circuit 814, LED light source 815, projection device 816, projection lens 817, external device connection I/F (Interface) 818, fan drive circuit 819, cooling A fan 820 is provided.

Among these, the CPU 801 controls the operation of the entire projector. The ROM 802 stores programs used to drive the CPU 801. RAM803 is used as a work area for CPU801. A media I/F 807 controls reading or writing (storage) of data to a recording medium 806 such as a flash memory.

The operation unit 808 is provided with various keys, buttons, LEDs, etc., and is used by the user to perform various operations other than turning on/off the power of the projector 150. For example, the operation unit 808 receives instruction operations such as a projection image size adjustment operation, a color tone adjustment operation, a focus adjustment operation, a keystone adjustment operation, and outputs the received operation contents to the CPU 801.

The power switch 809 is a switch for switching the power of the projector 150 ON/OFF. The bus line 810 is an address bus, a data bus, etc. for electrically connecting each component such as the CPU 801 shown in FIG. 3.

The network I/F 811 is an interface for data communication using a communication network such as the Internet. The LED drive circuit 814 controls turning on and off of the LED light source 815 under the control of the CPU 801.

When the LED light source 815 is turned on under the control of the LED drive circuit 814, it irradiates the projection device 816 with projection light. The projection device 816 modulates the projection light from the LED light source 815 using a spatial light modulation method based on the image data provided via the external device connection I/F 818, and outputs the modulated light through the projection lens 817. , and project it as an image onto the projection surface of the screen. As the projection device 816, for example, a liquid crystal panel or a DMD (Digital Micromirror Device) is used. The LED drive circuit 814, LED light source 815, projection device 816, and projection lens 817 collectively function as a projection unit (projection means) that projects a projection image onto a projection surface based on image data.

The external device connection I/F 818 is directly connected to a PC (Personal Computer) and acquires control signals and image data with the PC.

The fan drive circuit 819 is connected to the CPU 801 and the cooling fan 820, and drives/stops the cooling fan 820 based on a control signal from the CPU 801.

By rotating, the cooling fan 820 exhausts the air inside the projector 150 and cools the inside of the projector 150.

When power is supplied to the projector 150 from the power supply circuit 21, the projection device 816 enters an image display state, and power is supplied from the power supply circuit 21 to the other various components. When power is supplied from the power supply circuit 21 to the projector 150, the projection device 816 enters an image display state, and power is supplied from the power supply circuit 21 to the other various components.

Further, in the projector 150, when the power switch 809 is operated to turn off, a power OFF signal is sent from the power switch 809 to the CPU 801, and when the CPU 801 detects the power OFF signal, it gives a control signal to the LED drive circuit 814 to turn off the LED. The light source 815 is turned off. Thereafter, when a predetermined period of time has elapsed, the CPU 801 gives a control signal to the fan drive circuit 819 to stop the cooling fan 820, ends its own control processing, and finally gives an instruction to the power supply circuit 21 to turn off the power supply. Stop the power supply.

<<Information processing device>>
Fig. 4 is a hardware configuration diagram of the information processing device 110. As shown in Fig. 4, the information processing device 110 is constructed by a computer, and includes a CPU 501, a ROM 502, a RAM 503, a HD 504, a HDD (Hard Disk Drive) controller 505, a display 506, an external device connection I/F (Interface) 508, a network I/F 509, a bus line 510, a keyboard 511, a pointing device 512, a DVD-RW (Digital Versatile Disk Rewritable) drive 514, and a media I/F 516.

Of these, the CPU 501 controls the operation of the entire information processing device. The ROM 502 stores programs used to drive the CPU 501, such as IPL. The RAM 503 is used as a work area for the CPU 501. The HD 504 stores various data such as programs. The HDD controller 505 controls the reading or writing of various data from the HD 504 according to the control of the CPU 501. The display 506 displays various information such as a cursor, menu, window, character, or image. The external device connection I/F 508 is an interface for connecting various external devices. In this case, the external device is, for example, a USB (Universal Serial Bus) memory or a printer. The network I/F 509 is an interface for data communication using a communication network. The bus line 510 is an address bus, a data bus, or the like for electrically connecting each component such as the CPU 501 shown in FIG. 4.

The keyboard 511 is a type of input means equipped with multiple keys for inputting characters, numbers, various instructions, etc. The pointing device 512 is a type of input means for selecting and executing various instructions, selecting a processing target, moving the cursor, etc. The DVD-RW drive 514 controls the reading and writing of various data from the DVD-RW 513, which is an example of a removable recording medium. Note that this is not limited to a DVD-RW, and may be a DVD-R, etc. The media I/F 516 controls the reading and writing (storing) of data from the recording medium 515, such as a flash memory.

<Summary of amendment>
Hereinafter, an overview of the calculation process of various correction coefficients and the correction process based on the correction coefficients will be described with reference to FIGS.

<<Overall functional configuration>>
Fig. 5 is a functional block diagram of the projection system 100 according to this embodiment. The projection system 100 shown in Fig. 5 includes a plurality of functional blocks that operate on an information processing device 110. The information processing device 110 includes a content storage unit 112, correction processing units 114a to 114c for each projector, projection image output units 116a to 116c for each projector, and switching units 122a to 122c for each projector. The information processing device 110 further includes a calibration image storage unit 118, a calibration scene selection unit 120, a calibration captured image input unit 124, and a correction coefficient calculation unit 130.

The content storage unit 112 stores a file of a content image to be projected as a single projection image 106. The content storage unit 112 is configured as a storage area of a hard disk drive (HDD), a solid state drive (SSD), or a removable medium. The content image to be projected may be provided as a display screen when a file is executed in an application such as a word processor or presentation, or may be provided as a still image, or may be provided as a frame at any timing in a video file. The content image does not have to be provided as a file, and may be provided as a screen generated by the information processing device 110 by executing an operating system, or as a video externally input to the information processing device 110. For the sake of convenience, the following description will be given taking as an example a case where a content image is provided as a still image.

Correction processing units 114a to 114c are provided corresponding to projectors 150a to 150c included in projection system 100. The correction processing unit 114 reads each content image from the content storage unit 112, performs correction processing, and generates a corresponding projection image for the projector. Note that the details of the processing executed by each of the correction processing units 114a to 114c will be described later.

Projection image output units 116a to 116c are provided corresponding to projectors 150a to 150c included in projection system 100. Each projection image output unit 116 includes a display output connected to the corresponding projector 150, and outputs the input image selected by switching unit 122 to the connected projector.

The switching units 122a to 122c switch the flow of images according to the operation mode of the projection system 100. During a projection mode in which a content image is projected, the switching unit 122 switches the input side to the output of the correction processing unit 114. Accordingly, during the projection mode, the projection image output unit 116 outputs the processing result based on the content image by the corresponding correction processing unit 114 as a video. On the other hand, during the calibration mode, the switching unit 122 switches the input side to the output of the calibration scene selection unit 120, which will be described later. Accordingly, during the calibration mode, the projection image output unit 116 outputs the calibration images selected and output by the calibration scene selection unit 120, respectively.

The calibration image storage unit 118 stores a calibration image to be projected from the projector 150 during the calibration mode. The calibration image storage unit 118 is configured as a storage area of an HDD, SSD, or removable media. The calibration image is typically provided as a still image prepared in advance.

Here, the calibration image includes both or one of a lattice pattern that defines the lattice points of the projected image and an alignment pattern that defines the alignment points of the projected image. FIG. 6 is a diagram illustrating two types of calibration images used in the projection system 100 according to this embodiment. FIG. 6A illustrates a first calibration image 200 that includes both an alignment pattern 202 and a grid pattern 206. FIG. 6B illustrates a second calibration image 210 that includes only the alignment pattern 212.

The grid pattern 206 defines the coordinates on the projector memory, and is configured as a pattern in which arbitrary graphic elements are arranged according to a predetermined rule. By capturing an image of the grid pattern 206 projected onto the screen 102, it is possible to detect keystone distortion and local distortion of the projected image. In the first calibration image 200 illustrated in Fig. 6A, the grid pattern 206 is obtained by dividing the entire projection area of the projector 150 into 10 blocks horizontally and 7 blocks vertically, and in the central 8 x 5 blocks, white circles 204 are arranged in a grid pattern on a black background.

However, the grid pattern 206 is not particularly limited, and various patterns can be used, such as a polka dot pattern in which circles that contrast with the background are arranged two-dimensionally as shown in FIG. 6(A), a dot pattern in which dots that contrast with the background are arranged two-dimensionally, a checkerboard pattern in which squares of two colors that contrast with each other are arranged alternately in the horizontal and vertical directions, and a grid pattern in which lines that contrast with the background are arranged two-dimensionally. The method of dividing the entire projection area of the projector 150 is also not limited to the above-mentioned aspects, and the number of divisions of the areas and the method of dividing the areas can be determined depending on the required accuracy and the performance of the information processing device 110.

The alignment patterns 202, 212 define the reference positions (alignment points) of the projected images between the captured images, and are configured as a pattern of multiple arbitrary graphic elements arranged at predetermined positions. By capturing the common alignment patterns 202, 212 projected onto the screen 102 multiple times, alignment can be performed between the multiple captured images.

In the first calibration image 200 that includes both the alignment pattern and the grid pattern, the alignment pattern 202 preferably extends around the area where the grid pattern 206 is arranged, as shown in FIG. 6(A). It can be placed in any position. Even in the second calibration image 210 that includes only the alignment pattern, as shown in FIG. 6(B), it is placed at the same position as the calibration image shown in FIG. An alignment pattern 212 is arranged at a position around the region.

In the first calibration image 200 illustrated in FIG. 6A, the alignment pattern 202 has rectangular markers 202LT, 202RT, 202LB, and 202RB arranged at the four corners of the outer periphery of a 10×7 block of the entire projection image area of the projector 150. I am using the one that I made. Similarly, in the second calibration image illustrated in FIG. 6(B), the alignment pattern 212 uses rectangular markers 212LT, 212RT, 212LB, and 212RB arranged at the four corners of the outer periphery of a 10×7 block.

However, the alignment patterns 202 and 212 are not particularly limited. The shape of the markers of the alignment patterns 202 and 212 may be any graphic element such as a circle, and the number of markers may be at least four. Note that alignment accuracy can be improved by increasing the number of markers for alignment.

Now, let us refer to FIG. 5 again. In the calibration process according to this embodiment, the grid pattern is captured multiple times to detect the geometric distortion of the projected image of the projector 150, and the results of the multiple captures are integrated using the alignment pattern. The calibration scene selection unit 120 reads out each calibration image from the calibration image storage unit 118 described above, and selects and outputs an appropriate calibration image for each of the multiple projectors 150a to 150c. Here, the calibration scene selection unit 120 grasps the positional relationship of the projected images of the multiple projectors 150, and the calibration images are selected according to each stage of the calibration process so that the overall calibration results of each projector 150 are obtained without excess or deficiency. The scene at each calibration process stage, including the calibration image projected by at least one projector 150, is referred to as the calibration projection scene.

The calibration scene selection unit 120 causes each projector 150 to project a calibration image corresponding to the calibration projection scene. At this time, the user uses the camera 160 to capture each calibration projection scene so that the projected calibration projection image fits within the angle of view. The calibration captured image input unit 124 receives input of each captured image from the camera 160 via wireless connection, wired connection, or removable media, and prepares multiple calibration captured images for calibration processing. Note that for at least one calibration projection scene, the user is required to capture the image facing the screen, and typically, the first capture is performed facing the screen using a spirit level or the like. In this case, the second and subsequent captures do not need to be performed facing the screen. The results are integrated based on the calibration captured image captured facing the screen 102.

In an embodiment using three projectors 150a to 150c as shown in FIG. 2, calibration projection scenes can be constructed in two ways as shown in FIG. 7 and FIG. 8, and calibration captured images can be captured. FIG. 7 is a diagram illustrating the calibration projection scenes selected sequentially by the calibration scene selection unit 120 and the method of capturing them in the first way. FIG. 8 is a diagram illustrating the calibration projection scenes selected sequentially by the calibration scene selection unit 120 and the method of capturing them in the second way.

In the first mode, as shown in FIG. 7, three calibration projection scenes corresponding to three imaging operations are prepared. In the first calibration projection scene, the first projector 150a projects the first calibration image 200 shown in FIG. 6(A), and the second projector 150b projects the second calibration image 210 shown in FIG. 6(B). The third projector 150c does not project anything. In this first calibration projection scene, the camera 160 is used to capture images so that the projection images 230a and 230b of the first projector 150a and the second projector 150b are within the field of view.

In the second calibration projection scene, the first projector 150a does not project anything, the second projector 150b projects the first calibration image 200 shown in FIG. 6(A), and the third projector 150c projects the second calibration image 210 shown in FIG. 6(B). In the second calibration projection scene, the camera 160 is used to capture images so that the projection images 232b and 232c of the second projector 150b and the third projector 150c are within the field of view. In the last third calibration projection scene, the third projector 150c projects the first calibration image 200 shown in FIG. 6(A). The first projector 150a and the second projector 150b do not project anything. In this third calibration projection scene, the camera 160 is used to capture images so that the projection image 234c of the third projector 150c is within the field of view.

Note that in FIG. 7, three projection scenes for calibration are prepared corresponding to three projectors 150a to 150c that arrange three projection images in the horizontal direction. However, when generalized to N (N≧2) projectors 150, it is sufficient to configure N calibration projection scenes as described below. In other words, the n-th (1≦n≦N-1.)th calibration projection scene is such that one of the adjacent projectors (the n-th one) in the arrangement of projection images is as shown in FIG. 6(A). The first unit projects a calibration image including an alignment pattern (the first unit does not necessarily need to project an alignment pattern) and at least a grid pattern, and the other unit (n+1 unit) projects the alignment pattern as shown in FIG. 6(B). A calibration image is prepared to be projected that includes only the alignment pattern as shown. The final Nth calibration projection scene is prepared so that the last projector (Nth unit) projects a calibration image including both a grid pattern and an alignment pattern as shown in FIG. 6(A). be done. In the two calibration images, the results are integrated using the alignment pattern projected by the common projector 150.

In the first aspect described above, the range that the camera 160 can capture in one image capture is at most the projection range of two cameras, so even if the number N increases, the depth constraint on the screen 102 is relaxed. In turn, it is possible to accommodate a variety of screen arrangements. Note that the same applies to the case where the projected images are arranged in the vertical direction and the case where the projected images are arranged two-dimensionally in a single stroke.

In the second mode, two calibration projection scenes corresponding to two imaging operations are prepared as shown in Fig. 8. In the first calibration projection scene, the first projector 150a and the third projector 150c project the first calibration image 200 shown in Fig. 6(A), and the second projector 150b projects the second calibration image 210 including only the alignment pattern shown in Fig. 6(B).
In the second calibration projection scene, the second projector 150b projects the first calibration image 200 shown in Fig. 6A. The first projector 150a and the third projector 150c do not project anything. In the first calibration projection scene, the projection images 240a to 240c of the first projector 150a to the third projector 150c are captured by the camera 160 so as to fit within the field of view, and in the second calibration projection scene, the projection image 242b of the second projector 150b is captured by the camera 160 so as to fit within the field of view.

Note that in FIG. 8, two calibration projection scenes are prepared corresponding to three projectors 150a to 150c that arrange three projected images in the horizontal direction. Generalizing to N (N≧3) projectors 150, it is sufficient to configure two calibration projection scenes as described below. That is, in the first calibration projection scene, in the arrangement of projection images, those of the first group (for example, odd-numbered projectors) that are alternately selected among the plurality of projectors 150 perform the first calibration including at least a grid pattern. is configured to project an image for use. In the second calibration projection scene, in the array of projection images, those in the second group (for example, even-numbered cameras) that are adjacent to those in the first group are configured to project a grid pattern. Further, the first calibration projection scene and the second calibration projection scene are configured such that at least one common projector projects the alignment pattern.

More preferably, the second group (e.g., the even-numbered units) can be configured to project a second calibration image including only an alignment pattern in the first calibration projection scene, and to project a first calibration image including both a grid pattern and an alignment pattern in the second calibration projection scene.

In the second aspect described above, although the range that the camera 160 can capture in one image is wider, the number of times that images need to be captured is only two. Therefore, if the problem of the depth constraint on the screen 102 can be avoided, the calibration work is simplified. The same applies to the case where the projected images are aligned in the vertical direction. Also, in the above description, an image of the entire first and second groups is captured once each. However, in other embodiments, in order to reduce the required angle of view, the first and second groups may each be captured in separate images multiple times, and the images may be integrated via a common alignment pattern within the groups.

Referring now to FIG. 5 again. The correction coefficient calculation unit 130 reads out the plurality of calibration captured images prepared by the calibration captured image input unit 124, and calculates various correction coefficients to be set for the correction processing units 114a to 114b. Here, it is assumed that each calibration captured image and the above-mentioned calibration scene are provided to the calibration captured image input unit 124 in association with each other. In more detail, the correction coefficient calculation unit 130 includes a feature point extraction unit 132, a grid point conversion unit 134, a geometric correction coefficient calculation unit 136 that calculates a geometric correction coefficient, and a correction coefficient for blending of projection images. , a gradation changing section 139 , and an overlapping area detecting section 140 .

The feature point extraction unit 132 extracts feature points from each of the prepared multiple calibration images. The feature points extracted here may include lattice points that indicate distortion of the projected image and correspond to the lattice pattern of the calibration image, and alignment points that serve as a reference for alignment between the calibration imaging devices and correspond to the alignment pattern of the calibration image.

The lattice points for each projector and the alignment points for each captured image extracted by the feature point extraction unit 132 are passed to the lattice point conversion unit 134. The lattice point conversion unit 134 converts and integrates the lattice points (at this stage, the lattice points are points on the coordinate system of each calibration captured image) of the projection images of each projector 150 extracted from the multiple calibration captured images by the feature point extraction unit 132 onto a common coordinate system based on the alignment points common to the calibration captured images. In the embodiment described below, the common coordinate system is the coordinate system of the first calibration captured image captured facing the screen 102.

The lattice points for each projector transformed onto the common coordinate system by the lattice point transformation unit 134 are passed to the geometric correction coefficient calculation unit 136. Based on the lattice points on the common coordinate system, the geometric correction coefficient calculation unit 136 calculates a geometric correction coefficient for each projector that provides a projected image to be projected from each of the multiple projectors 150. The geometric correction coefficient is a correction coefficient that incorporates geometric corrections such as alignment, scale adjustment, and distortion correction.

The overlapping area detection unit 140 detects, for each of the plurality of projectors 150, an overlapping area between a projected image of a projector of interest (for example, 150a) and a projected image of each of the projectors (for example, 150b) adjacent to the projector of interest. To detect. The brightness adjustment unit 138 calculates a blending coefficient for adjusting the superposition of these projection images based on the detection result of the overlapping area. If the blending coefficient is less than 1, it has the effect of reducing the brightness of the overlapping area. Using this blending coefficient for each projector, images are smoothly combined on the screen 102 where the projected images of the plurality of projectors 150 overlap.

The correction processing unit 114 generates a projection image for each projector from the content image based on various correction coefficients calculated by the geometric correction coefficient calculation unit 136 and the brightness adjustment unit 138, respectively. More specifically, the correction processing unit 114 first generates an intermediate image for each projector from the content image based on the geometric correction coefficient calculated by the geometric correction coefficient calculation unit 136. The intermediate image is obtained by deforming the image handled by the content image projector in a manner opposite to the detected geometric distortion. The correction processing unit 114 then multiplies the intermediate image by the blending coefficient calculated by the brightness adjustment unit 138 to calculate a final projected image for each projector. During the projection mode, the switching unit 122 selects the projection image calculated by the correction processing unit 114 and outputs it to the projector 150.

The processing of this embodiment has been described as being realized by a single information processing device 110, but the embodiment of the projection system 100 is not limited to that shown in FIG. 5. In other embodiments, in order to reduce the load on the information processing device 110 due to an increase in the number of devices, the correction processing units 114a to 114c may be realized on each of the projectors 150a to 150c. In other embodiments, the processing described in this embodiment may be distributed and executed by multiple information processing devices 110, the functions of the information processing device 110 may be implemented on one of the projectors 150, or a single device may be configured that has the functions of the information processing device 110 shown in FIG. 5 and the functions of multiple projectors. Furthermore, in other embodiments, the functions of the correction coefficient calculation unit 130 may be implemented as a server that provides the functions of the correction coefficient calculation unit 130 as a service via a network.

<Overall processing flow>
Hereinafter, with reference to FIG. 9, the overall flow of calculation processing of various correction coefficients and correction processing based on the correction coefficients will be described. FIG. 9 is a flowchart showing the overall processing flow of calculation processing of various correction coefficients and correction processing based on the correction coefficients according to the present embodiment. The process shown in FIG. 9 starts from step S100 in response to a user's instruction to start the calibration process. Note that FIG. 9 shows both the first aspect shown in steps S101 to S104 and the second aspect shown in steps S105 to S108 together.

In the first aspect shown in FIG. 7, in step S101, the information processing device 110 causes the first projector 150a to output a first calibration image (including a grid pattern), and the second projector 150b to output a second calibration image. Output the image (including the alignment pattern). In step S102, the information processing device 110 causes the second projector 150b to output a first calibration image (including a grid pattern and an alignment pattern), and causes the third projector 150c to output a second calibration image (including an alignment pattern). ) is output. In step S103, the information processing device 110 causes the third projector 150c to output the first calibration image (including the grid pattern and alignment pattern). At each step from step S101 to step S103, the user, for example, follows the guidance provided by the information processing device 110 and captures an image so that the calibration image being projected falls within the angle of view of the camera 160. In step S104, the information processing device 110 collectively receives input of a plurality of captured images for calibration from the camera 160, and the process proceeds to step S109.

On the other hand, in the second mode shown in FIG. 8, in step S105, the information processing device 110 causes the first projector 150a and the third projector 150c to output the first calibration image (including the grid pattern), A second calibration image (including the alignment pattern) is output from the second projector 150b. In step S106, the information processing device 110 receives input from the camera 160 of the calibration image captured in step S105. In step S107, the information processing device 110 causes the second projector 150b to output the first calibration image (including the grid pattern and alignment pattern). In step S108, the information processing device 110 receives the calibration image captured in step S107 from the camera 160, and the process proceeds to step S109.

In step S109, the information processing device 110 calculates the geometric correction coefficients for each projector, as will be described in detail later. In the process of calculating the geometric correction coefficients for each projector in step S109, feature points are extracted from each calibration image, the grid points of each calibration image are converted to a common coordinate system, and the geometric correction coefficients for each projector are calculated. In step S110, the information processing device 110 calculates the blending coefficients for each projector, as will be described in detail later.

In step S111, the information processing device 110 sets the geometric correction coefficients and blending coefficients for each projector calculated in steps S109 and S110 for each correction processing unit 114. In step S112, the information processing device 110 causes the switching unit 122 to switch the input of the projection image output unit 116 to the output of the correction processing unit 114, and transitions to the projection mode.

In step S113, the information processing device 110 reads the content image. In step S114, the information processing device 110 performs a correction process on the content image using the correction processing unit 114 for each projector. In step S115, the information processing device 110 outputs the corrected projection images for each projector from the projection image output unit 116 for each projector.

In step S116, the information processing device 110 determines whether or not an instruction to end the projection mode has been received from the user. If it is determined in step S116 that an instruction to end the projection mode has not been received (NO), the process loops to step S113, and the projected image is updated. If it is a moving image, the process proceeds to processing the next frame. If it is determined in step S116 that an instruction to end the projection mode has been received (YES), the process branches to step S117, and the process ends.

<Calculation of geometric correction coefficient>
The calculation process of the geometric correction coefficients of each projector will be described in detail below with reference to Figs. 10 to 16 and Fig. 20(A). Fig. 10 is a flowchart showing the calculation process of the geometric correction coefficients executed by the correction coefficient calculation unit 130 according to this embodiment. The process shown in Fig. 10 is called in step S109 shown in Fig. 9, and starts from step S200.

In step S201, the feature point extracting unit 132 converts the centroid coordinates of the circular shape of each projected image of the projector 150 in each captured image coordinate system into lattice point coordinates (decimal point accuracy) from each of the plurality of prepared captured images for calibration. Extract as. The coordinates of the center of gravity of a circular shape can be calculated, for example, by binarizing the image, cutting out a cluster of white pixels by pattern matching, etc., and finding the coordinates of the center of gravity.

In step S202, the feature point extraction unit 132 extracts the barycentric coordinates of the rectangular markers of the projected images of the projectors 150 in each of the captured image coordinate systems from each of the plurality of calibration captured images as alignment point coordinates. Similarly, the coordinates of the center of gravity of a rectangular marker can be calculated by, for example, binarizing the image, cutting out a cluster of white pixels by pattern matching, etc., and finding the coordinates of the center of gravity.

More specifically, the first aspect shown in FIG. 7 will be described. From each, in the arrangement of projected images, a positioning point of a positioning pattern (the first projector is not necessarily projected) projected by one of the adjacent projectors (nth projector) and at least a grid point of a grid pattern; The alignment points of the alignment pattern projected by the other (n+1)th unit are extracted. The grid points of the grid pattern projected by the last projector (Nth projector) and the alignment points of the alignment pattern are extracted from the captured calibration image obtained by capturing the last Nth calibration projection scene. .

In addition, in one captured image for calibration, the circular pattern of one projector and the four alignment rectangular markers of the other projector can be identified using their positional relationship. When the projection scene for calibration is configured in the first mode shown in FIG. , two rectangular markers on the left side of the right projector, two rectangular markers on the right side of the left projector, and two rectangular markers on the right side of the right projector, in this order. Based on such positional relationships, each circular pattern and rectangular marker can be identified. In addition to the positional relationship, for example, by changing the color and shape of the rectangular marker to be projected for each projector, the color and shape of the imaged rectangular marker can be identified, and the discrimination can be performed based on these identified characteristics. You may go.

To explain the second aspect shown in FIG. 8, the feature point extracting unit 132 extracts the features of the first group ( For example, the lattice points of the lattice pattern projected by the odd-numbered unit are extracted. From the captured calibration image obtained by capturing the second calibration projection scene, grid points of the grid pattern projected by the second group (for example, even-numbered units) are extracted. The feature point extracting unit 132 further extracts the alignment points of the alignment pattern projected by the common projector 150 from each of the calibration images obtained by capturing the first calibration projection scene and the second calibration projection scene. Extract.

In step S203, the lattice point conversion unit 134 calculates a projective transformation coefficient for a predetermined pair of calibration captured images based on the alignment point coordinates of the rectangular marker common between the captured images. In step S204, the lattice point conversion unit 134 converts and integrates the lattice point coordinates of the projected images of each projector into a common coordinate system based on the projection transformation coefficient calculated in step S203.

FIG. 11 is a diagram illustrating three calibration captured images prepared by capturing each calibration projection scene and projective transformation coefficients calculated between these captured images in the first embodiment. FIG. 12 is a diagram illustrating two calibration captured images prepared by capturing each calibration projection scene and projective transformation coefficients calculated between these captured images in the second embodiment.

In the first aspect, as shown in FIG. 11, for a pair of the first calibration image 250 and the second calibration image 260, a pair of alignment point coordinates of the rectangular markers 254, 264 of the projection images 252b, 262b of the second projector 150b that are common between the captured images is obtained. Based on this pair of alignment point coordinates (254, 264), the lattice point conversion unit 134 calculates a coefficient of projective transformation that converts the coordinate system of the second calibration image 260 to the coordinate system of the first calibration image 250. Similarly, for a pair of the second calibration image 260 and the third calibration image 270, a pair of alignment point coordinates of the rectangular markers 264, 274 of the projection images 262c, 272c of the third projector 150c that are common between the captured images is obtained. Based on this pair of alignment point coordinates (264, 274), the coefficients of the projective transformation that converts from the coordinate system of the third calibration image 270 to the coordinate system of the second calibration image 260 are calculated.

The transformation formula for projective transformation is expressed by the following formula (1), and by subtracting the denominator from the formula (1) and rearranging it, it can be expanded into a first-order polynomial expressed by the following formula (2).

上記式（１）及び（２）中、ｘ，ｙは、変換前の平面座標を表し、ｕ，ｖは、変換後の平面座標を表し、ａ～ｈの８つの係数は、射影変換係数を表す。上記式において、未知パラメータである８つの射影変換係数を算出するためには、最低でも８個の連立方程式を要するが、上述した２枚の校正用撮像画像における４つの位置合わせの対応点があれば、８個の変換式を生成することができる。この４つの矩形マーカの対応点から生成された８つの連立方程式を解くことにより、射影変換係数ａ～ｈを求めることができる。

In the above equations (1) and (2), x and y represent the plane coordinates before transformation, u and v represent the plane coordinates after transformation, and the eight coefficients a to h represent the projective transformation coefficients. represent. In the above equation, at least eight simultaneous equations are required to calculate the eight projective transformation coefficients that are unknown parameters. For example, eight conversion formulas can be generated. By solving eight simultaneous equations generated from the corresponding points of these four rectangular markers, the projective transformation coefficients a to h can be determined.

In the first aspect, when the projective transformation coefficients a to h between the two pairs of captured images are calculated, the lattice point conversion unit 134 converts the extracted lattice points of the second calibration captured image into the first A projective transformation is performed to transform the captured image into the coordinate system of the captured image. The lattice point conversion unit 134 further performs projective transformation to convert the extracted lattice points of the third calibration captured image from the coordinate system of the third calibration captured image to the coordinate system of the second calibration captured image. , and further performs projective transformation for converting the coordinate system of the second captured image for calibration to the coordinate system of the first captured image for calibration. As a result, the grid point coordinates of all the projectors 150a to 150c are converted and integrated into a common coordinate system, which is the coordinate system of the first calibration image taken directly facing the screen.

In the second mode, as shown in FIG. 12, for a pair of a first calibration captured image 280 and a second calibration captured image 290, a pair of alignment point coordinates of rectangular markers 284, 294 of projected images 282b, 292b of the second projector 150b that are common between the captured images is obtained. Then, based on this pair of alignment point coordinates, a projective transformation coefficient that converts from the coordinate system of the second calibration captured image 290 to the coordinate system of the first calibration captured image 280 is calculated. The lattice point conversion unit 134 performs projective transformation that converts the extracted lattice points of the second calibration captured image into the coordinate system of the first captured image based on the projective transformation coefficients a to h between a pair of calibration captured images, and converts them into a common coordinate system.

Figure 13 is a diagram that shows a schematic of a set of lattice point coordinates of each projector integrated on a common coordinate system 300. As shown in Figure 13, the sets of lattice point coordinates 302a, 302b, and 302c of each projector are each transformed and integrated on the common coordinate system 300 of the first calibration image. Note that Figure 13 shows the circular shapes of the captured projectors 150a-150 superimposed, but the images themselves do not need to be superimposed.

Referring again to FIG. 10, in step S205, the geometric correction coefficient calculation unit 136 linearly extrapolates each of the lattice point coordinates that have been transformed and integrated onto the common coordinate system for each projector 150, and calculates the outer perimeter coordinates of the projectable area.

FIG. 14 is a diagram showing a method of calculating the outer circumferential coordinates of the projectable area by linear extrapolation using each integrated grid point coordinate. FIG. 14(A) shows the four grid points in the upper left corner on the projector memory, and FIG. 14(B) shows the corresponding four grid points on the common coordinate system. As shown in FIG. 14(A), the outer circumferential coordinates (lattice points on the four corners and four sides) on the projector memory are a quadrilateral patch of four grid points (for example, P00P to P11P) located on the outer circumference. It is determined at the position to be extrapolated (a position at a distance 1.5 times the distance between grid points).

The coordinates of the outer perimeter pixels (grid points on the four corners and four sides) corresponding to the projectable area of each projector in the common coordinate system can be calculated by linearly extrapolating from the coordinates of the four lattice points located on the perimeter, as shown in Figure 14 (B). Similarly, a point on the common coordinate system corresponding to any coordinate point on the projector memory other than the outer perimeter coordinates (grid points on the four corners and four sides) can be found by linearly interpolating or extrapolating the grid point coordinates of four nearby points.

Here, assume that an arbitrary coordinate point _QP on the projector memory is a point that is divided internally by t:1-t (0 < t < 1) in the x-axis direction and s:1-s (0 < s < 1) in the y-axis direction at four neighboring lattice points _P00P , _P10P , _P01P , and _{P11P, whose} coordinate positions on the projector memory are close to each other. Then, a point QC on the common coordinate system corresponding to the coordinate point QP can be calculated using the following formula (3) from the coordinate vectors of the corresponding four lattice points _P00C , _P10C , _P01C , and _P11C . In the case of a point to be extrapolated, the calculation can be performed using the following formula (3) by setting the ranges of -1.5 < t < 0, -1.5 < s < 0 for the above t and s.

画像全体では非線形な幾何歪みが生じ得るが、ここでは、その一部である２×２の格子点で構成される四辺形パッチの範囲及び外周に向かって所定量外挿した範囲では、線形な幾何歪みだと見なせることを前提としている。上記四辺形パッチのサイズが充分に小さいとみなせるためである。なお、説明する実施形態では、上記式（３）を用いて線形補間により対応点を計算するものとして説明する。しかしながら、他の実施形態では、隣接する４つの格子点のＰ００_Ｃ，Ｐ１０_Ｃ，Ｐ０１_Ｃ，Ｐ１１_Ｃ、Ｐ００_Ｐ，Ｐ１０_Ｐ，Ｐ０１_Ｐ，Ｐ１１_Ｐを用いて求められる射影変換により、プロジェクタ・メモリ上の点Ｑ_Ｐを共通座標系上での対応点Ｑ_Ｃに対応付けてもよい。

Although nonlinear geometric distortion may occur in the entire image, it is assumed here that the distortion is linear in the range of a quadrilateral patch consisting of 2x2 lattice points, which is a part of the image, and in the range extrapolated by a predetermined amount toward the periphery. This is because the size of the quadrilateral patch can be considered to be sufficiently small. In the embodiment described below, the corresponding points are calculated by linear interpolation using the above formula (3). However, in other embodiments, the point QP on the projector memory may be associated with the corresponding point _QC on the common coordinate _system by projective transformation obtained using the four adjacent lattice points _P00C , _P10C , _P01C , _P11C , _P00P , _P10P , _P01P , and _P11P .

By performing the above-mentioned linear extrapolation for each projector, the projectable areas of the three projectors 150a to 150c (i.e. the range in which an all-white image can be projected) are detected on a common coordinate system. Figure 15(A) shows the projectable areas 304a to 304c of the three projectors detected on a common coordinate system 300. The projectable area 304a of the first projector 150a is shown by a solid white line, the projectable area 304b of the second projector 150b is shown by a dashed white line, and the projectable area 304c of the third projector 150c is shown by a two-dot chain white line.

Referring again to FIG. 10, in step S206, the geometric correction coefficient calculation unit 136 calculates the logical sum (OR) of the projectable areas of all the projectors in the common coordinate system, and calculates the corrected value for mapping the content image. A projection target area is set within the area of the above-mentioned logical sum. The corrected projection target area is set so that the content image is mapped at its maximum size within an area that is the logical sum of the projectable areas 304a to 304c of all projectors while maintaining its aspect ratio.

The four corner points of each projectable area on the common coordinate system are known, and the four sides (top, bottom, left, and right sides) that connect them are also found linearly divided by the grid point width. The range of existence of is also grasped. Therefore, in the range sandwiched by the upper side 306T and lower side 306B of the projectable areas 304a to 304c of the three projectors on the common coordinate system, and also in the range sandwiched by the left side 306L and right side 306R, there are three logical sum areas. The rectangular range that can be taken within is determined.

The corrected projection target area 310 is an area allocated at the maximum size while maintaining the aspect ratio (e.g. M:N) of the content image within the rectangular range of the four sides 306T, 306B, 306L, and 306R, as shown by the dotted lines in Figure 15 (A). In the example of Figure 15 (A), there is some vertical space, so margins are provided above and below and the corrected projection target area is centered. Then, as shown in Figure 15 (B), the content image 320 to be projected is pasted into this corrected projection target area 310.

Referring again to FIG. 10, in the loop of steps S207 to S211, the processes shown in steps S208 to S210 are executed for each projector, and the geometric correction coefficients for each of the multiple projectors are calculated. In step S208, the geometric correction coefficient calculation unit 136 converts the lattice point coordinates on the common coordinate system into the coordinate system of the original content image. Hereinafter, the content image pasted onto the post-correction projection target area 310 on the common coordinate system is referred to as the "projected content image", and the original content image that is the source of this is referred to as the "full-size content image".

In step S209, the geometric correction coefficient calculation unit 136 maps the grid point coordinates on the projector memory to pixel positions in the coordinate system of the full-size content image via the common coordinate system. In step S210, the geometric correction coefficient calculation unit 136 maps the integer pixel coordinates on the projector memory to pixel positions in the coordinate system of the full-size content image via the common coordinate system by linear interpolation.

The geometric correction coefficients calculated in each process shown in steps S208 to S210 correspond to each coordinate in the projector memory 330 with a pixel position on the full-size content image that corresponds to the position on the projected content image, as shown in FIG. 16.

Taking one lattice point _P42P in the projector memory 330a shown in Fig. 16 as a representative, a corresponding point _P42C ( _XP42C , _YP42C ) on the common coordinate system 300 is extracted for the lattice point _P42P on the projector memory 330. Then, since the content image is mapped to the post-correction projection target area 310, a corresponding pixel position _P42m ( _XP42m , _YP42m ) on the life-size content image is further determined for the corresponding point _P42C on the common coordinate system 300, as shown in Fig. 16.

The corresponding pixel position _P42m ( _XP42m , _YP42m ) on the same-size content image is calculated from the coordinates ( _XP42C , _YP42C ) of the corresponding point _P42C on the common coordinate system 300 using the following formula (4). can do. In the following equation (4), the coordinates (X ₀ , Y ₀ ) are the coordinates of the upper left origin of the projected content image on the common coordinate system, and R represents the scaling factor of the content image. Here, for convenience of explanation, it is assumed that the same-size content image is directly mapped to the corrected projection target area 310 at a predetermined magnification ratio R, but the method of mapping the content onto the common coordinate system is particularly It is not limited.

プロジェクタ・メモリ上の格子点Ｐ４２_Ｐ以外のすべての格子点Ｐｉｊ_Ｐ（ｉｊは整数）についても同様に、等倍コンテンツ画像上での対応する画素位置が計算される。プロジェクタ・メモリ上の格子点以外の任意の座標については、図１４を参照して説明した同様の方法により、近傍の２×２格子点のコンテンツ画像上での対応する画素位置を線形に補間（ 内挿又は周辺部では外挿）することによって、等倍コンテンツ画像上での対応する画素位置を計算することができる。これにより、プロジェクタ・メモリ３３０ａ上の所定領域３３２ａの画素に対し、コンテンツ画像３２０における第１プロジェクタ１５０ａが担当する領域３２２ａの画素位置が対応付けられる。

Similarly, for all lattice points Pij _P (ij is an integer) other than lattice point P42 _P on the projector memory, the corresponding pixel positions on the full-size content image are calculated. For any coordinate other than the lattice points on the projector memory, the corresponding pixel positions on the full-size content image can be calculated by linearly interpolating (interpolating or extrapolating in the periphery) the corresponding pixel positions on the content image of nearby 2×2 lattice points using the same method as described with reference to Fig. 14. In this way, the pixel positions of the area 322a in the content image 320 assigned to the first projector 150a are associated with the pixels of the predetermined area 332a on the projector memory 330a.

Figure 20 (A) shows an example of the data structure of the geometric correction coefficient for one projector calculated in the processing of steps S208 to S210. As shown in Figure 20 (A), the corresponding pixel positions on the life-size content image for all pixels in the projector memory calculated in this way become the geometric correction coefficient.

The loop from step S207 to step S211 is repeated for the number of projectors, and when the correspondence between the integer pixel coordinates on the projector memory and the coordinate system of the same-sized content image is completed for all projectors, the process proceeds to step S212. In step S212, this process is ended and the process is returned to the calling source shown in FIG. As a result, geometric correction coefficients for all projectors 150a to 150c are prepared.

In the embodiment described, the geometric correction coefficients are obtained by calculating the pixel positions on the life-size content image corresponding to all pixels in the projector memory, but the present invention is not limited to this. In another embodiment, the geometric correction coefficients may be calculated by calculating pixel positions Pij _m on the life-size content image corresponding to each lattice point Pij _P on the projector memory, and the correction processing unit 114 described later may calculate the coordinates other than the lattice points by performing projective transformation or linear transformation for each quadrilateral patch.

<Calculation of blending coefficient>
The details of the calculation process of the blending coefficient of each projector will be described below with reference to FIGS. 17 to 20. FIG. 17 is a flowchart showing a blending coefficient calculation process executed by the correction coefficient calculation unit 130 according to the present embodiment. The process shown in FIG. 17 is called in step S110 shown in FIG. 9 and starts from step S300. In the loop of steps S301 to S313, the processes shown in steps S302 to S312 are executed for each projector of interest, and the blending coefficients of each of the plurality of projectors 150a to 150c are determined.

In step S302, the overlapping area detection unit 140 detects overlapping areas of the target projector and adjacent projectors of the target projector based on the outer periphery coordinates of the projectable areas of the target projector and adjacent projectors in the common coordinate system 300. FIG. 18 is a diagram illustrating the correspondence of blending coefficients to each coordinate on the projector memory 330. As shown in FIG. 18, at the top edge of the corrected projection target area 310 in the common coordinate system 300, by searching from the left origin (○) to the right, the first projector 150a and the second projector 150b are The starting point (●) and ending point (◎) of the overlapping area are detected. That is, when searching from the left origin (◯) to the right, the overlapping area is from the projectable area 304b of the second projector 150b to the projectable area 304a of the first projector 150a. The start point and end point of the overlapping area are similarly detected for other horizontal lines.

Referring again to FIG. 17, in step S302, the brightness adjustment unit 138 first initializes the blending coefficient for each coordinate in the common coordinate system to 0. In the loop of steps S304 to S311, the processes shown in steps S305 to S310 are executed for each horizontal line in the common coordinate system (only the portion corresponding to the post-correction projection target area). Through the processes shown in steps S305 to S310, intermediate blending coefficient results are assigned to each coordinate position on the common coordinate system.

In step S305, in the horizontal line of interest, based on the above-mentioned outer circumferential coordinates of the projectable area and the detected overlapping area, the start and end points of the projectable area of the projector and the overlapping area with the adjacent projector are determined. Set the start and end points.

In the loop of steps S306 to S310, the processes shown in steps S307 to S309 are executed for each pixel (only within the projectable area) on the horizontal line of the common coordinate system. The processes shown in steps S307 to S309 determine the blending coefficient for each pixel on the common coordinate system on the horizontal line.

In step S307, the brightness adjustment unit 138 branches the process depending on whether the pixel of interest corresponds to an overlapping area. If it is determined in step S307 that the area does not correspond to an overlapping area (NO), the process proceeds to steps S308A and S308B. In this case, since it is a non-overlapping area, the gradation changing unit 139 changes the RGB range to a limited range (16 to 235) (if the original RGB image is in a limited range, no change is necessary). Further, in step S308B, the brightness adjustment unit 138 determines the blending coefficient to be the maximum value 1. The details of changing the gradation will be explained with reference to FIG. 22.

On the other hand, if it is determined in step S307 that the area corresponds to an overlapping area (YES), the process proceeds to steps S309A and S309B. In this case, since it is an overlapping area, the gradation changing unit 139 changes the RGB range to the full range (0 to 255) (if the original RGB image is in the full range, no change is necessary). Further, in step S309B, the brightness adjustment unit 138 calculates a blending coefficient according to a predetermined relational expression.

FIG. 19 shows a graph of the input/output characteristics of a projector, but the input/output characteristics of a projector are typically not linear. In calculating the blending coefficient for the pixel corresponding to the above-mentioned overlapping area, the input/output characteristics are reversely corrected to make them linear, and then the light amounts from the projectors on both sides add up to 1. Perform weighting.

Specifically, as shown for the first projector in the graph at the bottom of FIG. 18, for pixels in the range from the origin (○) to the start point (●) of the overlapping area, the blending coefficient is determined to be a maximum of 1 in step S308. On the other hand, for pixels in the range from the start point (●) to the end point (◎) of the overlapping area, a blending coefficient is calculated in step S309 by applying an inverse correction to the input/output characteristics of the projector so that the actual brightness gradually drops linearly from 1.0 to 0 depending on the horizontal distance from the start point (●). With input/output characteristics as shown in FIG. 19, the blending coefficient y for the horizontal distance x (0.0≦x≦1.0) from the start point normalized by the distance from the start point to the end point can be calculated using the following formula (5).

ステップＳ３０４～ステップＳ３１１のループにより、共通座標系における整数画素各々に対しブレンディング係数の中間結果が決定される。投影可能領域以外の領域については、ステップＳ３０３の初期化により０が設定されている。ステップＳ３０４～ステップＳ３１１のループで、共通座標系のすべての水平ラインについての処理が完了すると、ステップＳ３１２へ処理が進められる。補正後投影目標領域以外の水平ラインについては、ステップＳ３０３の初期化により各画素に０が設定されている。

An intermediate result of the blending coefficient is determined for each integer pixel in the common coordinate system by the loop of steps S304 to S311. For areas other than the projectable area, 0 is set by the initialization of step S303. When the processing for all horizontal lines in the common coordinate system is completed by the loop of steps S304 to S311, the processing proceeds to step S312. For horizontal lines other than the post-correction projection target area, 0 is set for each pixel by the initialization of step S303.

In step S312, the brightness adjustment unit 138 associates, for each integer pixel coordinate in the projector memory, a blending coefficient assigned to the nearest integer pixel of the coordinate (decimal point) in the common coordinate system, which is associated by the data structure shown in FIG. 20(A). FIG. 20(B) illustrates an example of the data structure of the blending coefficient of one projector calculated by the processing of steps S302 to S312. As shown in FIG. 20(B), the blending coefficients for all pixels in the projector memory are obtained.

When the process for all projectors is completed in the loop from step S301 to step S313, the process is ended in step S314, and the process is returned to the calling source shown in FIG.

By the above process, blending coefficients for all pixels in the projector memory are obtained for each of the multiple projectors 150a to 150c. Note that the above explanation is about the overlapping area between the first projector 150a and the second projector 150b. When the second projector 150b is the target, the blending correction calculation is performed for the two overlapping areas, combining the left and right first projector 150a and the third projector 150c.

<Gradation correction>
FIG. 21 is a diagram for explaining gradation correction in the projection system 100. In a projection system using three projectors, the projected images are arranged so that they overlap in part to avoid gaps. A projection image is transmitted from the information processing device 110 to each projector, for example, via a network. Each projector stores the transmitted image in a RAM or SSD, and projects the image when an instruction to start projection is transmitted. In the case of a moving image, the information processing device 110 synchronizes the projectors 150a to 150c for projection. For example, the information processing device 110 receives projection completion notifications from all projectors and then instructs the projection of the next projection image. Synchronization may be achieved by communication between the projectors 150a to 150c.

Figure 22 is a diagram explaining gradation correction from full range to limited range. Full range refers to a gradation expression method in which RGB signals are expressed in 256 gradations from 0 to 255 (in the case of 8 bits), and limited range refers to a gradation expression method in which RGB signals are expressed in 220 gradations from 16 to 235 (in the case of 8 bits). Generally, processing is done in full range inside a computer. PC monitors generally process in full range. Television sets were originally limited range, but regular televisions have the function of converting limited range to full range.

FIG. 22A is a diagram schematically showing gradation conversion of non-overlapping areas when the RGB range of the original image data is a full range. Since the non-overlapping area is converted to a limited range, the gradation becomes narrower. Conversion from a full range RGB signal to a limited range RGB signal is performed by equally associating the full range with the limited range. Since "256-220=36", as shown in FIG. 22(b), RGB values in the full range are associated with the same RGB values in the same limited range approximately once every seven gradations. FIG. 22(b) is a diagram showing an example of a correspondence table. With such a correspondence table, a full range RGB signal is converted to a limited range. Note that the example of FIG. 22(b) in which the correspondence is made equal is one example, and the full range and the limited range are matched one-to-one in at least one of the dark area (for example, less than 16) and the bright area (for example, more than 235). Alternatively, full range RGB values may be associated with the same limited range RGB values in the intermediate region.

Note that the gradation of the overlapping area remains full range, so if the RGB range of the original image data is full range, no conversion is necessary.

For example, an RGB value of 10 in the full range of a non-overlapping area is converted to 25. An RGB value of 10 in the overlapping area is not converted. As a result, 25>10, the black level of the non-overlapping area is raised, and the overlapping area is projected darker than the non-overlapping area. Because the overlapping area is not corrected to be darker, it is less likely to be crushed.

As explained above, image data handled within a computer is primarily full range, but there are also cases where the image data before conversion is limited range. When the RGB signal is limited range (16 to 235), the gradation conversion unit 139 leaves the RGB range of the non-overlapping area as the limited range. Then, it converts the RGB range of the overlapping area to full range.

FIG. 23 is a diagram illustrating tone correction from limited range to full range. FIG. 23(a) is a diagram schematically showing gradation conversion in an overlapping area when the original RGB signal is in a limited range. Since the overlapping area is converted to a full range, the gradation becomes wider. Conversion from limited range RGB to full range RGB is performed by equally associating the limited range with the full range. Therefore, the conversion from full range to limited range is the opposite. As shown in FIG. 23(b), the RGB values of the limited range are associated with different full ranges approximately once every seven gradations. For example, if the RGB value is 22, it will be associated with the full range of 6 or 7 once every two times. FIG. 23(b) is a diagram showing an example of a correspondence table. With such a correspondence table, limited range RGB signals are converted to full range signals. Note that the example of FIG. 23(b) in which the correspondence is made evenly is one example, and the limited range and the full range are made to correspond one-to-one in at least one of the dark area (for example, less than 16) or the bright area (for example, more than 235), In the intermediate region, limited range RGB values may be associated with different full range RGB values.

For example, in the limited range, an RGB value of 25 in a non-overlapping area remains unconverted at 25. An RGB value of 25 in an overlapping area is converted to 10. As a result, 25>10, and the overlapping area is projected darker than the non-overlapping area. The overlapping area has a wider range of gradations, so it is less likely to be crushed blacks.

After tone correction, a blending process adjusts the overlapping areas of adjacent projectors so that they are equivalent to the non-overlapping areas, preventing the overlapping areas from becoming unnaturally bright.

Although the present embodiment has been described using the terms full range and limited range, it is only necessary that the overlapping area has a larger gradation than the non-overlapping area. Therefore, for example, the RGB range of the non-overlapping area may be set to 50-200, and the RGB range of the overlapping area may be set to 0-255. Alternatively, the RGB range of the non-overlapping area may be set to 50 to 255, and the RGB range of the overlapping area may be set to 0 to 255.

<Correction processing>
Hereinafter, details of the correction process based on the above correction coefficient will be explained with reference to FIGS. 20 and 24. FIG. 24 is a diagram illustrating a correction process based on the above correction coefficient. The geometric correction coefficient of each projector calculated by the geometric correction coefficient calculation unit 136 described above and the blending coefficient of each projector calculated by the brightness adjustment unit 138 described above are calculated in step S111 shown in FIG. section 114.

First, the correction processing unit 114 prepares association data between all pixels in the projector memory and corresponding pixel positions on the same-size content image. When the pixel positions for all pixels of the projector memory as shown in FIG. 20(A) have been determined by the processing in the geometric correction coefficient calculation unit 136 described above, the correction processing unit 114 calculates the pixel positions as shown in FIG. 20(A). The mapping data shown in is read out as is. If only the pixel position on the same-size content image for each grid point coordinate of the projector memory is given, all pixels on the projector memory other than the grid point can be calculated by linear interpolation etc. from the coordinates of the grid point. Coordinates on the same-size content image to be referenced are calculated, and correspondence data as shown in FIG. 20(A) is calculated.

The correction processing unit 114 generates an intermediate image from the life-size content image to be projected by a pixel interpolation method such as bilinear or bicubic, based on the pixel position (decimal point number) on the life-size content image to be referenced for each pixel in the projector memory. The correction processing unit 114 further multiplies the pixel values of R, G, and B of the generated intermediate image by blending coefficients associated with each other by the association data in FIG. 20(B), to generate the final projected image.

FIG. 24 shows projected images 350a to 350c finally obtained from content images by the correction processing units 114a to 114c for three projectors 150a to 150c. As shown in FIG. 24, these projection images 350a-350c are projected from the projector 150 during the projection mode. Since the projected image 350 has been subjected to various corrections on the portion of the content image that the corresponding projector 150 is responsible for, the projected images 350a to 350c are suitably overlapped on the projection plane and are displayed in a single image. The images are combined into one projected image 352.

<Major Effects>
As described above, the projection system of this embodiment can prevent the overlapping areas from becoming unnaturally bright while suppressing crushed blacks by making the gradation of the overlapping areas wider than that of the non-overlapping areas.

<Other application examples>
The above describes the best mode for carrying out the present invention using examples, but the present invention is not limited to these examples in any way, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

For example, in the configuration example shown in FIG. 5, the information processing apparatus 110 is divided according to main functions in order to facilitate understanding of the processing performed by the information processing apparatus 110. The present invention is not limited by the method of dividing the processing units or the names thereof. The processing of the information processing device 110 can also be divided into more processing units depending on the processing content. Furthermore, one processing unit can be divided to include more processing.

Furthermore, each function of the embodiments described above can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and devices such as an ASIC (Application Specific Integrated Circuit), DSP (digital signal processor), FPGA (field programmable gate array), and conventional circuit modules designed to execute each function described above.

100 Projection System 110 Information Processing Device 150 Projector

Patent No. 3908255

Claims (5)

A projection system in which an information processing device transmits image data to each of a plurality of projection devices, and each of the plurality of projection devices projects an image onto a projection body,
The information processing device includes:
an overlapping area detection unit that detects an overlapping area of the adjacent images from captured images of images projected on the projector;
a gradation changing unit that changes the gradation of the image of the overlapping area detected by the overlapping area detection unit to a full range of 0 to 255, and changes the gradation of the image of the non-overlapping area to a limited range of 16 to 235 ;
a brightness adjustment unit that reduces the brightness of the overlapping area of the image whose gradation has been changed by the gradation change unit;
a projection image output unit that transmits image data in which the brightness adjustment unit has reduced the brightness of the overlapping area to a plurality of projection devices;
The projection system is characterized in that the plurality of projection devices receive the image data transmitted by the projection image output section, and each projects an image. The projection system according to claim 1, characterized in that, when the gradation of the image before the gradation change is full range, the gradation change unit does not change the gradation of the image in the overlapping area, and changes the gradation of the image in the non -overlapping area to a limited range of 16 to 235. The projection system according to claim 1, wherein the gradation changing unit changes the gradation of the image in the overlapping area to a full range when the gradation of the image before the gradation change is in a limited range, and does not change the gradation of the image in the non - overlapping area. A projection method performed by a projection system in which an information processing device transmits image data to a plurality of projection devices, and the plurality of projection devices each project an image onto a projection object, comprising:
The information processing device includes:
an overlapping area detection unit detecting an overlapping area between adjacent images from a captured image of the image projected onto the projection body;
a gradation changing unit changing the gradation of the image of the overlapping area detected by the overlapping area detection unit to a full range of 0 to 255 and changing the gradation of the image of the non-overlapping area to a limited range of 16 to 235 ;
A brightness adjustment unit reduces the brightness of the overlapping region of the image whose gradation has been changed by the gradation change unit;
a step of a projection image output unit transmitting image data in which the brightness adjustment unit has reduced the brightness of the overlapping region to a plurality of projection devices;
a step of the plurality of projection devices receiving the image data transmitted by the projection image output unit and each projecting an image;
13. A projection method comprising: An information processing device that sends image data to multiple projection devices,
an overlapping area detection unit that detects an overlapping area of the adjacent images from the captured images of the images projected on the projector;
a gradation changing unit that changes the gradation of the image of the overlapping area detected by the overlapping area detection unit to a full range of 0 to 255, and changes the gradation of the image of the non-overlapping area to a limited range of 16 to 235 ;
a brightness adjustment unit that reduces the brightness of the overlapping area of the image whose gradation has been changed by the gradation change unit;
A program for causing the brightness adjustment unit to function as a projection image output unit that transmits the image data with reduced brightness of the overlapping area to a plurality of projection devices.

Images