JP4479820B2 - Image processing apparatus, image processing method, and imaging apparatus - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and an imaging apparatus. Specifically, by detecting a region of the presentation image presented by the presentation device from the captured image and performing gamma correction for each of the detected region of the presentation image and the other regions other than the presentation image, This makes it possible to obtain a captured image in which collapse is prevented.

  In recent years, it has been frequently performed to connect a computer device to a projector and project an image such as a document on a screen for presentation. Along with this, in order to save the contents of seminars and meetings as minutes, there are increasing opportunities to take pictures of presentations using an imaging device such as a video camera. Further, a captured image obtained by imaging a screen is displayed on a remote monitor device, and a remote conference or the like is performed, for example. Furthermore, not only images projected on a screen but also a monitor of a computer device and a display screen of a television device are often photographed.

  As described above, when a presentation image such as a document or an image projected on the screen is captured, the presentation image in the captured image may be overexposed or blackout may occur in another image (hereinafter referred to as “background image”). There is a fear. The main cause of these problems is that there is a large difference in brightness between the presented image portion and other image portions.

FIG. 18 shows a case where an imaging device captures an image of a presentation, for example.
For example, when the screen is photographed with a composition as shown in FIG. 18A, adjustment is made in accordance with a dark background, and the presentation image projected on the screen is overexposed as shown in FIG. Arise. When the presentation image is overexposed in this way, the content of the presentation image cannot be read even if a presentation is captured for minutes, for example.

  In addition, when imaging is performed around the presentation image projected on the screen as shown in FIG. 19, adjustment is performed in accordance with the presentation image, so that the background portion where the presenter is present is a darkened image. This makes it difficult to see the facial expressions and actions of the presenter.

  Therefore, for example, in the invention of Patent Document 1, the same subject is imaged under different exposure conditions, and a plurality of obtained images are divided into appropriate exposure areas and inappropriate exposure areas based on a predetermined image signal level, The improper exposure area applies image data of the proper exposure area in the image captured under other exposure conditions, performs gradation correction of this appropriate exposure area on each image, and synthesizes each image into one wide dynamic range. An image is generated.

JP 2000-228747 A

  By the way, when a wide dynamic range image is generated by synthesizing images as in the invention of Patent Document 1, different exposure conditions are used in order to make the synthesized image simple and the image without unnaturalness. The position of the presentation image must be fixed in the captured image captured in (1). For this reason, if a presentation image is captured in a state where the imaging device is not fixed, or if the presentation image is updated in the middle of imaging under different exposure conditions, a wide dynamic range image without unnaturalness is obtained. It becomes difficult.

  Therefore, the present invention provides an image processing apparatus, an image processing method, and an imaging apparatus that can easily obtain a good captured image in which a subject is prevented from being overexposed or darkened.

  The concept of the present invention is that when a presentation image presented by a presentation device, for example, an image projected on a screen by a projector or an image displayed on a screen of a television device is captured, the presentation image and the background image are captured from the captured image. In this case, a captured image in which overexposure and underexposure are prevented in each of the presentation image and the background image is obtained.

  The image processing apparatus according to the present invention includes a presentation image detection unit that detects a region of a presentation image presented by the image presentation device from a captured image based on a difference in luminance or color in the captured image, and a detection in the captured image. A gamma correction unit configured to perform gamma correction with different characteristics for each of the displayed image area and other areas excluding the displayed image.

  Further, an image processing method according to the present invention includes a presented image detection step of detecting a region of a presented image presented by an image presentation device from a captured image based on a difference in luminance or color in the captured image, It has a gamma correction step of performing gamma correction with different characteristics for the detected area of the presentation image and other areas excluding the presentation image.

  Furthermore, an imaging apparatus according to the present invention includes: an imaging unit that captures an image of a subject to obtain a captured image; and a region of a presentation image that is presented from the captured image by the image presentation device based on a difference in luminance or color in the captured image. The present invention includes a presentation image detection unit that performs detection, and a gamma correction unit that performs gamma correction with different characteristics on the detected presentation image region and other regions other than the presentation image in the captured image.

  In the present invention, a region of the presentation image is detected from the captured image, and gamma correction is performed with different characteristics on the detected region of the presentation image and the background image region excluding the presentation image. For example, gamma correction is performed on the background image area with characteristics corrected to expand the luminance level of the low luminance portion. In addition, gamma correction is performed on the region of the presentation image with characteristics corrected to compress the luminance level of the high luminance part and characteristics corrected to enhance the contrast.

  According to the present invention, regions of the presentation image and the background image are detected from the captured image, and gamma correction is performed with different characteristics for each region. For this reason, it is possible to easily obtain a good captured image in which overexposure or blackout of the subject is prevented.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 shows the configuration of the imaging apparatus. The subject light is incident on the imaging element 211 of the imaging unit 21 such as a charge coupled device (CCD) image sensor or a complementary metal oxide semiconductor (CMOS) image sensor via the lens unit 11 and the aperture mechanism 12.

  The imaging element 211 converts the imaging light of the subject image formed on the light receiving surface into an electrical signal for each pixel to generate an image signal of three primary colors. Further, the generated image signal is supplied to a CDS (Correlated Double Sampling circuit) unit 212.

  The CDS unit 212 removes noise from the image signal supplied from the image sensor, and supplies the image signal after the noise removal to the amplification unit 213. The amplification unit 213 adjusts the gain of the image signal supplied from the CDS unit 212 and supplies the image signal Sa after gain adjustment to the A / D (Analog / Digital) conversion unit 23.

  The timing signal generator 22 generates a drive signal TD for generating an image signal by the image sensor 211 and supplies the drive signal TD to the image sensor 211. The timing signal generation unit 22 controls the exposure time (effective charge accumulation time obtained by photoelectric conversion) in the image sensor 211 by the drive signal TD.

  The A / D converter 23 converts the analog image signal Sa into a digital image signal DVa and supplies the digital image signal DVa to the presentation image detector 31, the detector 32, and the image processor 35.

  The presentation image detection unit 31 performs processing for detecting a region of the presentation image from the captured image based on the image signal DVa, generates a detection signal JA indicating the detection result, and supplies the detection signal JA to the image processing unit 35 and the control unit 51.

  The detection method of the presentation image is not limited to a specific detection method, and any detection method may be used as long as the region of the presentation image can be accurately detected from the captured image. For example, the presentation image detection unit 31 converts the three primary color image signals DVa into a luminance signal DY and a color difference signal DC, and binarizes the luminance signal DY by comparing the luminance signal DY with a threshold value. Generates a digitized signal BY. The threshold used for binarization may be set to a level fixed in advance, or may be set according to the captured image so that the presented image can be detected from the captured image with high accuracy. . Next, the presentation image detection unit 31 obtains a horizontal distribution by counting pixels in which the signal level of the luminance signal DY is higher than a threshold value for each line. Similarly, for each pixel position in the horizontal direction, the vertical distribution is obtained by counting in the vertical direction pixels whose signal level of the luminance signal DY is higher than the threshold value. Further, the presented image detection unit 31 detects the upper side and the lower side from the horizontal distribution and the left side and the right side from the vertical distribution. For example, the presentation image detection unit 31 detects the position of the edge from the position where the rising (falling) of the horizontal distribution and the vertical distribution has occurred, the position where the rising (falling) has been completed, or the like. Next, when the presented image detection unit 31 determines that the detected upper side, lower side, left side, and right side correspond to each side of the presented image, for example, it is surrounded by the detected upper side, lower side, left side, and right side. When it is determined that the area is approximate to the aspect ratio of the presented image, the detected area JA surrounded by the detected upper side, lower side, left side, and right side is set as the presented image area, and the detection signal JA indicating the presented image area is This is supplied to the image processing unit 35 and the control unit 51.

  In addition, when the presentation image detection unit 31 detects the vertex of the image based on the horizontal distribution and the vertical distribution and determines that the detected vertex corresponds to each vertex of the presentation image, for example, the detected vertex When it is determined that the enclosed area approximates the aspect ratio of the presented image, the area enclosed by connecting the detected vertices with a straight line may be used as the presented image area. In addition, the presentation image detection unit 31 uses the position where the rising of the horizontal distribution and the vertical distribution has occurred and the position where the rising of the distribution is completed as a vertex.

  Moreover, the presentation image detection part 31 is good also as what detects an edge and detects the area | region of a presentation image from an edge detection result. That is, an edge that is a boundary between the black level and the white level is detected from the luminance binarized signal. Next, a boundary line indicating the boundary between the black level region and the white level region is detected. Here, since the presented image is rectangular, in detecting the boundary line, a linear boundary line is detected. Furthermore, an area corresponding to the presentation image is selected from a rectangular or trapezoidal area surrounded by the detected boundary line. Further, the presentation image detection unit 31 can extract four contours by extracting the contour, and can also set a region surrounded by connecting the vertices with a straight line as a region of the presentation image.

  The detection unit 32 generates a detection signal JB indicating the brightness of the captured image from the supplied image signal DVa and supplies the detection signal JB to the control unit 51. For example, the detection unit 32 divides the captured image into m × n detection blocks, and calculates the sum of signals indicating the brightness of each pixel in each block. Further, the detection unit 32 weights the calculated sum for each block, and supplies a detection signal JB indicating an added value of the weighted sum to the control unit 51.

  The image processing unit 35 performs various image processing on the image signal DVa supplied from the A / D conversion unit 23. For example, the white balance (WB) adjustment unit 351 performs white balance adjustment so that a white subject in the captured image is correctly displayed as white. The gamma correction unit 352 performs image signal level conversion using a preset gamma curve. Further, the gamma correction unit 352 performs gamma correction with different characteristics on the image area determined to be the presentation image and the image area determined to be the background image based on the detection signal JA from the presentation image detection unit 31. For example, gamma correction is performed so that overexposure does not occur in the area of the image determined to be the presentation image, and gamma correction is performed so that blackout does not occur in the area of the image determined as background image. .

  Further, the image processing unit 35 converts, for example, an image signal of three primary colors subjected to white balance adjustment, gamma correction, and the like into an image signal such as a luminance signal or a color difference signal according to an image encoding technique of a still image or a moving image. Then, compression / decompression processing is performed, and the obtained still image or moving image compression-encoded signal GDa is output to the recording / reproducing unit 37. Furthermore, a process of outputting an image signal and / or a compression-coded signal of a still image or a moving image that has not been subjected to compression / decompression processing to an external device (not shown) is performed. When the compressed encoded signal GDb is supplied from the recording / reproducing unit 37, the image processing unit 35 performs processing to return the compressed encoded signal GDb to the image signal before the compression encoding process. Further, the image processing unit 35 generates a display image signal HG from the image signal DVa and supplies it to the display unit 38. The image processing unit 35 also generates a display image signal HG in which information indicating that the presentation image capturing mode is set is displayed on the screen based on the display signal HE supplied from the control unit 51. .

  The recording / reproducing unit 37 records the compressed encoded signal GDa supplied from the image processing unit 35 on a recording medium such as a semiconductor memory, an optical disk, or a magnetic tape. Further, the compressed encoded signal GDb recorded on the recording medium is read and supplied to the image processing unit 35.

  The display unit 38 is configured using, for example, a liquid crystal display element or the like, and performs image display based on the display image signal HG supplied from the image processing unit 35.

  The lens driving unit 41 drives a focus lens (not shown) of the lens unit 11 so that the subject image is focused on the light receiving surface of the image sensor 211. Further, the zoom lens (not shown) of the lens unit 11 is driven so that the size of the subject image formed on the light receiving surface of the image sensor 211 becomes a desired size.

  The aperture driving unit 42 drives the aperture mechanism 12 to adjust the amount of light of the subject image formed on the light receiving surface of the image sensor 211.

  A user interface unit 52 is connected to the control unit 51. The user interface unit 52 includes an operation key and an interface for inputting a remote operation signal from the outside. The user interface unit 52 generates an operation signal PS corresponding to a user operation and supplies the operation signal PS to the control unit 51.

  The control unit 51 includes a CPU (Central Processing Unit) 511, a ROM (Read Only Memory) 512, a RAM (Random Access Memory) 513, and the like. The CPU 511 executes a program stored in the ROM 512, generates a control signal corresponding to the operation signal PS, and supplies the control signal to each unit, thereby controlling the operation of the imaging device according to a user operation. To do. The RAM is used as a working area for temporarily storing information when performing various controls.

  Further, the control unit 51 switches the imaging mode based on the detection signal JA supplied from the presentation image detection unit 31, supplies the control signal CTe corresponding to the imaging mode to the image processing unit 35, and responds to the imaging mode. Switch the gamma correction operation.

  The control unit 51 generates at least one of the control signals CTb, CTc, and CTd so that the detection value indicated by the detection signal JB supplied from the detection unit 32 becomes a preset reference value. . The control unit 51 supplies the generated control signal CTb to the aperture drive unit 42, the control signal CTc to the timing signal generation unit 22, and the control signal CTd to the amplification unit 213, so that the aperture of the aperture mechanism 12 is adjusted so as to achieve proper exposure. Setting, adjustment of exposure time in the image sensor 211, and gain adjustment of the image signal Sa are performed. Further, the control unit 51 generates a control signal CTa and supplies it to the lens driving unit 41 to drive the focus lens so that the subject image is focused on the light receiving surface of the image sensor 211. Causes processing to be performed.

  Next, the operation of the imaging apparatus will be described. FIG. 2 is a flowchart showing the gamma correction operation. In step ST1, the control unit 51 causes the presentation image detection unit 31 to perform presentation image detection and proceeds to step ST2. The presentation image detection unit 31 detects a region of the presentation image in the captured image when the presentation image presented by the presentation device is captured, generates a detection signal JA indicating the detection result, and the image processing unit 35 and the control unit. 51.

  FIG. 3 is a flowchart showing the presentation image detection operation. In FIG. 3, the upper side and the lower side are detected from the horizontal distribution, and the left side and the right side are detected from the vertical distribution, and it is determined that the detected upper side, lower side, left side and right side correspond to each side of the presented image. The case where the region of the presented image is detected is illustrated.

  In step ST101, the presentation image detection unit 31 performs a conversion process. The presentation image detection unit 31 generates a luminance signal DY and a color difference signal DC from the three primary color image signals DVa, and proceeds to step ST102.

  In step ST102, the presentation image detection unit 31 performs binarization processing. The presented image detection unit 31 binarizes the luminance signal DY and the color difference signal DC, generates a luminance binarized signal and a color difference binarized signal, and proceeds to step ST103.

  In the binarization process, as described above, the luminance signal DY and the color difference signal DC are binarized by comparing the luminance signal DY and the color difference signal DC with the threshold value, and the luminance binarized signal BY and the color difference binarization are performed. A signal BC is generated. The threshold used for the binarization process may be set to a fixed level in advance, or the threshold may be set according to the captured image so that the region of the presentation image can be accurately detected from the captured image. Good.

  Here, when setting a threshold value according to a captured image, for example, the threshold value is set using a statistical method. As a statistical method, a mode method, a discriminant analysis method, or the like is used. In the mode method, the brightness level histogram of the captured image has bimodality, and when the portion of the presented image is separated from the portion of another image (hereinafter referred to as “background image”), the valley position is set as a threshold value. It is a method to set as. The discriminant analysis method is a method in which the histogram is divided into two groups according to the luminance level, and the division level that maximizes the variance between the groups is set as a threshold value.

  In step ST103, the presented image detection unit 31 selects the luminance binarized signal BY and proceeds to step ST104.

  In step ST104, the presentation image detection unit 31 acquires a horizontal distribution and proceeds to step ST105. In step ST105, the presentation image detection unit 31 acquires the vertical distribution and proceeds to step ST106.

  FIG. 4 shows acquisition processing of the horizontal distribution and the vertical distribution. 4A shows a captured image, and FIG. 4B shows a captured image after binarization processing. The presentation image detection unit 31 obtains a horizontal distribution shown in FIG. 4C by counting pixels in which the signal level of the luminance signal DY is higher than the threshold value for each line. Further, by counting pixels in which the signal level of the luminance signal DY is higher than the threshold value in the vertical direction, a distribution in the vertical direction orthogonal to the horizontal direction is obtained as shown in FIG.

  In step ST <b> 106, the presented image detection unit 31 performs processing for detecting the upper side and the lower side of the presented image, and proceeds to step ST <b> 107.

  5 and 6 show the processing for detecting the upper and lower sides of the presented image. 5A shows a captured image after binarization processing, and FIG. 5B shows the horizontal distribution acquired in step ST104. Here, the presentation image detection unit 31 divides the horizontal distribution into a plurality of regions, and obtains the sum of target pixels for each region. For example, the region is divided into regions REh1, REh2, REh3, REh4, and REh5, and the sum of target pixels is obtained for each region. Next, the presented image detection unit 31 sequentially selects areas from the upper end to the lower end of the captured image, compares the sum of the target pixels in the selected area with a threshold value, and detects an area where the sum exceeds the threshold value. By performing such processing, it is possible to detect a region including the upper side of the presented image. Similarly, the presented image detection unit 31 sequentially selects areas from the lower end to the upper end of the captured image, compares the sum of the target pixels in the selected area with the threshold, and detects an area where the sum exceeds the threshold. By performing such processing, it is possible to detect a region including the lower side of the presented image.

  By the way, when the side of the presentation image is close to the boundary of the region, the sum of the target pixels in the region is greatly different, and it may be difficult to detect the region including the side of the presentation image. For example, if the upper side of the presentation image is in the region REh (n) and close to the region REh (n + 1), the sum of the target pixels in the region does not become a large value. Here, if a region where the sum exceeds the threshold value is detected, there is a possibility that the upper side of the presented image is erroneously determined to be included in the region REh (n + 1). Therefore, when the horizontal distribution is divided into a plurality of regions, the regions are provided so as to have portions overlapping with other regions. For example, regions are provided as shown in regions REh1 to REh5 and REh1s to REh4s. In this case, even if the upper side of the presented image is near the boundary between the region REh1 and the region REh2, and the position is in either the region REh1 or the region REh2, the total sum of the target pixels in the region REh1s does not vary greatly. Therefore, if the threshold is set so that the upper side of the presentation image is included in the region REh1s when the total sum of the target pixels in the region REh1s at this time is compared with the threshold, the region including the side of the presentation image Can be detected correctly. The threshold value may be fixed in advance, or may be set from the sum of target pixels in each region using a statistical method as described above.

  Next, in the region REh1s including the side of the detected presentation image, for example, the region REh1s including the upper side, the presented image detection unit 31 sequentially calculates the sum of the target pixels for each line in the horizontal distribution for each line from the upper end to the lower end of the captured image. The line whose sum exceeds the threshold is used as the upper side La of the presentation image. Similarly, in the region REh4s including the lower side, the presented image detection unit 31 sequentially uses the sum of the target pixels for each line in the horizontal distribution from the lower end to the upper end of the captured image for each line and compares it with the threshold value. A line exceeding the threshold is set as the lower side Lb of the presentation image. 6A shows a captured image after binarization processing, and FIG. 6B shows a horizontal distribution. Here, when the region REh1s is determined as the region including the upper side, the sum of the target pixels for each line in the region REh1s is sequentially used for each line from the upper end to the lower end of the captured image and compared with the threshold value. A line exceeding the upper limit La is set as the upper side La of the presentation image. Similarly, when the region REh4s is determined as the region including the lower side, the sum of the target pixels for each line in the region REh4s is sequentially used for each line from the lower end to the upper end of the captured image and compared with the threshold value. A line that exceeds is set as the lower side Lb of the presentation image.

  In this way, even if the total of one line becomes large due to noise or the like by detecting the area including the upper side and the lower side and detecting the upper side and the lower side of the presentation image from the detected area, this line Can be prevented from being erroneously determined as the upper side of the presented image.

  In step ST107, the presentation image detection unit 31 performs detection processing of the left side and the right side of the presentation image, and proceeds to step ST108.

  7 and 8 show the detection processing of the left and right sides of the presented image. FIG. 7A shows the captured image after the binarization process, and FIG. 7B shows the vertical distribution acquired in step ST105. Here, the presentation image detection unit 31 divides the vertical distribution into a plurality of regions, for example, regions REv1 to REv6 and REv1s to REv5s, and obtains the sum of the target pixels for each region. Next, the presented image detection unit 31 sequentially selects areas from the left end to the right end of the captured image, compares the sum of the target pixels in the selected area with a threshold value, and detects an area where the sum exceeds the threshold value. By performing such processing, the region REv2 including the left side of the presented image can be detected. Similarly, the presented image detection unit 31 sequentially selects regions from the right end to the left end of the captured image, compares the sum of the target pixels in the selected region with a threshold value, and detects a region where the sum exceeds the threshold value. By performing such processing, the region REv5 including the right side of the presented image can be detected. The threshold value at this time may be fixed in advance, or may be set from the sum of target pixels in each region using a statistical method as described above.

  Next, the presented image detection unit 31 sequentially uses the sum of the target pixels for each pixel position in the vertical distribution in the detected region, for example, the region REv2 including the left side, sequentially from the left end to the right end of the captured image for each pixel position. In comparison, the pixel position where the sum exceeds the threshold value is defined as the left side Lc of the presentation image. Similarly, in the region REv5 including the right side, the presented image detection unit 31 sequentially uses the sum of the target pixels for each pixel position in the vertical distribution for each pixel position in the direction from the right end to the left end of the captured image and compares it with the threshold value. The pixel position where the sum exceeds the threshold is defined as the right side Ld of the presented image.

  FIG. 8A shows a captured image after binarization processing, and FIG. 8B shows a vertical distribution. Here, when the region REv2 is determined as a region including the left side, the sum of the target pixels for each pixel position in the region REv2 is sequentially used from the left end to the right end of the captured image and compared with the threshold, and the sum exceeds the threshold. Let the pixel position be the left side Lc of the presentation image. Similarly, when the region REv5 is determined as the region including the right side, the sum of the target pixels for each pixel position in the region REv5 is sequentially used for each line from the right end to the left end of the captured image and compared with the threshold value. A line exceeding the threshold is set as the right side Ld of the presentation image.

  In step ST108, the presentation image detection unit 31 determines whether or not the presentation image is present. The presented image detection unit 31 determines the aspect ratio from the upper and lower sides and the left and right sides detected in steps ST106 and ST107, and this aspect ratio is the aspect ratio of the presented image, for example, the aspect ratio “used in television broadcasting”. When approximate to "4: 3", "16: 9", or the like, that is, when the determined aspect ratio is "4: 3 ± α", "16: 9 ± β", in steps ST106 and ST107 The area surrounded by the detected sides is determined as a presentation image, the presentation image detection process is terminated, and the process proceeds to step ST2 in FIG. “Α” and “β” are constants indicating an allowable range for determining that the image is a presentation image, and are set in advance. In this way, by determining whether or not the aspect ratio approximates the desired aspect ratio, it is possible to easily determine the area of the presentation image from the captured image when a presentation image with a predetermined aspect ratio is captured. It becomes possible to do.

  In addition, when the presented image detection unit 31 cannot detect the upper and lower sides and the left and right sides of the presented image, the aspect ratio determined based on the detected upper and lower sides and the left and right sides approximates the desired aspect ratio. If not, the process proceeds to step ST109 assuming that the presented image is not detected.

  FIG. 9 is a diagram for explaining an operation of determining whether or not the image is a presentation image. For example, as shown in FIGS. 9A and 9B, an upper side La and a lower side Lb, a left side Lc and a right side Ld are detected, and an image surrounded by the detected upper side, lower side, left side and right side is presented. When the aspect ratio of the image is not approximated, the process proceeds to step ST109.

  In step ST109, the presentation image detection unit 31 determines whether or not the detection of the region of the presentation image is the first time. When the presentation image detection unit 31 detects the region of the presentation image using the luminance binarized signal, the presentation image region is detected for the first time, and the process proceeds to step ST110.

  In step ST110, the presentation image detection unit 31 selects the color difference binarized signal, returns to step ST104, and repeats the processing from step ST104.

  The presentation image detection unit 31 detects the presentation image based on the color difference binarized signal by repeating the processing from step ST104. The presented image detection unit 31 cannot detect the upper and lower sides and the left and right sides of the presented image, or the aspect ratio determined based on the detected upper and lower sides and the left and right sides approximates the desired aspect ratio. If not, the process proceeds from step ST108 to step ST109.

  In this way, if the presentation image is detected using not only the luminance binarized signal but also the color difference binary signal, for example, the luminance difference between the presented image and the background image is small. Even when it is difficult to distinguish between the presentation image and the background image based on this, when the presentation image is an image based on a color different from the background image, the presentation image can be detected from the difference in color. This presented image has a feature that it has a larger luminance value and a darker color than natural scenery. Here, the color density can be expressed using the magnitude of the color difference component in the color space of the YCC space or Lab space or the magnitude of the saturation in the HSV space. Therefore, if the presentation image is detected using the intensity of the color difference component and the magnitude of the saturation, the presentation image and the background image can be detected even if it is difficult to distinguish the presentation image and the background image based on the luminance difference. It becomes possible to classify, and a presentation image can be detected with high accuracy.

  In step ST109, the presentation image detection unit 31 determines whether or not the detection of the region of the presentation image is the first time. The presentation image detection unit 31 detects the region of the presentation image using the luminance binarization signal and the color difference binarization signal, and terminates the processing since the detection of the region of the presentation image is not the first time. The process proceeds to step ST2 of step 2.

  The detection of the region of the presentation image is not limited to the processing shown in FIG. 3, and other detection methods may be used as described above. For example, by performing edge detection of the captured image, a contour indicating the boundary between the presentation image and the background image is acquired, and straight lines indicating four vertices and four sides of the presentation image are obtained from the contour information, and the presentation image is obtained. Detect the area.

  In step ST2 of FIG. 2, the control unit 51 determines whether or not the condition for switching to the presentation image capturing mode is satisfied. When the control unit 51 determines that the condition for switching to the presentation image imaging mode is satisfied, the control unit 51 sets the imaging mode to the presentation image imaging mode and proceeds to step ST3. The imaging mode (for example, normal imaging mode) is set and the process proceeds to step ST5.

  The control unit 51 determines whether or not the condition for switching to the presentation image capturing mode is satisfied according to the generation state of the frame in which the presentation image is detected. For example, the control unit 51 counts up every time a frame in which a presentation image is detected is generated, and counts down every time a frame in which a presentation image is not detected occurs. Note that the range of the count value is set to “0” to “about several tens”. Here, when the count value is equal to or greater than a threshold value set in advance for determining the imaging mode (hereinafter referred to as “mode determination threshold value”), the presentation image imaging mode is set and the count value is less than the mode determination threshold value If so, set to normal imaging mode.

  Moreover, the control part 51 is good also as what sets the imaging mode based on the continuous number of the frames from which the presentation image was detected, or the continuous number of the frames from which the presentation image was not detected. For example, when the number of consecutive frames in which the presentation image is detected reaches a predetermined number, the imaging mode is switched from the normal imaging mode to the presentation image imaging mode. When the number of consecutive frames in which no presentation image is detected reaches a predetermined number, the imaging mode is switched from the presentation image imaging mode to the normal imaging mode. In this way, it is possible to stably switch the imaging mode according to the detection state of the presented image.

  In step ST3, the control unit 51 performs a presentation image capturing mode notification process and proceeds to step ST4. The control unit 51 notifies the user that the imaging device is set to the presentation image imaging mode. For example, when the presentation image capturing mode is set, the control unit 51 generates the display signal HE and supplies the display signal HE to the image processing unit 35 to generate the display image signal HG from the image signal DVa and the display signal HE. Thus, information indicating that the captured image and the presentation image capturing mode are set is displayed on the screen of the display unit 38. FIG. 10 shows a case where an icon display MH indicating that the presentation image imaging mode is set is provided on the captured image displayed on the display unit 38.

  In step ST4, the control unit 51 classifies the regions of the presentation image and the background image, causes the gamma correction unit 352 of the image processing unit 35 to perform gamma correction with different characteristics for each region, and proceeds to step ST7. . That is, the control unit 51 notifies the gamma correction unit 352 that the presentation image capturing mode has been set by the control signal CTe. The gamma correction unit 352 performs gamma correction with different characteristics for each region so that the presentation image and the background image do not cause overexposure or underexposure. The gamma correction unit 352 performs gamma correction using different characteristics that are corrected to compress the luminance level of the high-luminance portion and characteristics that are corrected to increase the luminance of the low-luminance portion.

  For example, when a presentation image such as an image projected on a screen or an image displayed on a television device is captured, the presentation image is generally brighter than the background image. Prone to occur. In addition, since the background image generally has a lower luminance than the presented image, the background image is likely to be blacked out in the captured image. Therefore, the gamma correction unit 352 divides the captured image into regions of the presentation image and the background image based on the detection signal JA, and corrects the luminance level of the high-luminance portion for the region of the presentation image. Perform gamma correction with. In addition, the gamma correction is performed on the background image area with characteristics corrected so as to extend the luminance level of the low luminance portion.

  FIG. 11 illustrates the region A of the presentation image and the region B of the background image in the captured image indicated by the detection signal JA from the presentation image detection unit 31. FIG. 12 shows correction characteristics (gamma curve) when the gamma correction unit 352 performs gamma correction. When performing gamma correction of the presented image, the gamma correction unit 352 compresses the luminance level of the high luminance portion as compared to the normal gamma curve CBa (shown by a broken line) as shown in FIG. A corrected characteristic gamma curve CBb (shown by a solid line) is used. Further, when performing gamma correction on the background image, the gamma correction unit 352 extends the luminance level of the low-luminance portion as compared to the normal gamma curve CBa (shown by a broken line), as shown in FIG. A gamma curve CBc (shown by a solid line) having the characteristics corrected as described above is used.

  When the process proceeds from step ST2 to step ST5, the control unit 51 performs a normal imaging mode notification process and proceeds to step ST6. The control unit 51 notifies the user that the imaging apparatus is set to the normal imaging mode. For example, when the display image HE is set in the presentation image capturing mode, the control unit 51 supplies the display signal HE to the image processing unit 35 to perform icon display MH indicating that the presentation image capturing mode is set. The supply of the signal HE is stopped and the icon display MH is deleted, so that the user can identify that the normal imaging mode is set.

  In step ST6, the control unit 51 causes the gamma correction unit 352 to perform gamma correction without dividing the region of the presentation image and the background image, and proceeds to step ST7. That is, the control unit 51 notifies the gamma correction unit 352 that the normal imaging mode has been set by the control signal CTe. When the normal imaging mode is set, the gamma correction unit 352 performs gamma correction using the gamma curve CBa that is not corrected to compress the luminance level of the high luminance portion or to expand the luminance level of the low luminance portion. .

  In step ST7, the control unit 51 determines whether or not the imaging operation is finished. The control unit 51 ends gamma correction when, for example, a power-off operation of the imaging apparatus 10 is performed or an operation of switching from the imaging mode to the reproduction mode for reproducing the recorded image is performed. When the power-off operation or the switching operation to the reproduction mode is not performed, the process returns to step ST1.

  FIG. 13 shows an operation when gamma correction is performed. Based on the detection signal JA, the gamma correction unit 352 divides the captured image shown in FIG. 13A into a presentation image shown in FIG. 13B and a background image shown in FIG. In FIG. 13B, the hatched portion indicates the background image region, and in FIG. 13C, the hatched portion indicates the presentation image region.

  The gamma correction unit 352 performs gamma correction on the region of the presentation image shown in FIG. 13B using the gamma curve CBb shown in FIG. At this time, since the gamma curve CBb has a characteristic of compressing the luminance level of the high luminance portion, the presentation image after the gamma correction is an image in which overexposure is prevented. Further, the gamma correction unit 352 performs gamma correction of the background image region shown in FIG. 13C using the gamma curve CBc shown in FIG. At this time, since the gamma curve CBc has a characteristic of extending the luminance level of the low luminance portion, the background image after the gamma correction is an image in which blackout is prevented. Therefore, as shown in FIG. 13D, the captured image that has been subjected to gamma correction by the gamma correction unit 352 can obtain a captured image that has no overexposure in the presentation image and no blackout in the background image.

  In this way, if the gamma correction is performed with different characteristics for the area of the presentation image and the area of the other image excluding the presentation image, excellent imaging in which overexposure and underexposure of the subject are prevented is performed. Images can be easily obtained.

  Incidentally, the gamma correction unit 352 performs gamma correction using the gamma curve obtained by compressing the luminance level of the high luminance part and the gamma curve obtained by extending the luminance level of the low luminance part as described above. When the brightness control is performed according to one of the presentation image and the background image, gamma correction may be performed with characteristics corrected only for the region of the other image.

  FIG. 14 is a flowchart showing an operation in the case of performing gamma correction by changing the gamma curve only in the background image region. In step ST11, the control unit 51 causes the presentation image detection unit 31 to perform presentation image detection in the same manner as in step ST1, and proceeds to step ST12. The presentation image detection unit 31 detects a region of the presentation image in the captured image when the presentation image presented by the presentation device is captured, generates a detection signal JA indicating the detection result, and the image processing unit 35 and the control unit. 51.

  In step ST12, the control unit 51 determines whether or not the switching condition to the presentation image capturing mode is satisfied, similarly to step ST2. When the control unit 51 determines that the condition for switching to the presentation image capturing mode is satisfied, the control unit 51 sets the imaging mode to the presentation image capturing mode and proceeds to step ST13. The imaging mode (for example, normal imaging mode) is set and the process proceeds to step ST16.

  In step ST13, the control unit 51 performs a presentation image capturing mode notification process in the same manner as in step ST3, and proceeds to step ST14.

  In step ST14, the control unit 51 fixes the brightness control and proceeds to step ST15. The control unit 51 fixes the brightness control by keeping the control signals CTb, CTc, and CTd constant regardless of the detection signal JB supplied from the detection unit 32.

  Some projectors that project an image onto a screen have a large light output or a small light output. However, a projector having a large light output is used for presentations in a large space such as a lecture hall, and an image is projected on a substantially entire surface of a large screen from a remote position. On the other hand, when an image is projected on a substantially entire surface of a large screen from a remote position with a projector having a small light output, the projected image becomes dark. For this reason, a projector having a small light output is used when an image is projected onto a small screen from a close position. Therefore, even when projectors with different light outputs are used, if the image is projected with a brightness that is easy to see, the brightness of the presented image is substantially equal. Thus, since the brightness of the presented image is substantially equal, the control signals CTb, CTc, CTd are set in advance so that the presented image in the captured image has a desired brightness, and this setting is performed. By using the control signals CTb, CTc, and CTd that are set to the presentation image imaging mode, a captured image in which the presentation image has a desired brightness can be obtained.

  In step ST15, the control unit 51 classifies the region of the presentation image and the background image, and causes the gamma correction unit 352 of the image processing unit 35 to perform gamma correction in which only the correction characteristics for the background image region are switched, and causes the step ST19 to move on. In other words, the control unit 51 notifies the gamma correction unit 352 that the presentation image capturing mode has been set by the control signal CTe. The brightness control of the gamma correction unit 352 is fixed so that the brightness of the presentation image is optimized in step ST14. Therefore, for the region of the presentation image, the luminance level of the high luminance part is compressed or the low luminance part is reduced. Gamma correction is performed using a gamma curve in which the luminance level is not expanded. For the background image area, gamma correction is performed using a gamma curve in which the luminance level of the low luminance portion is expanded. When the brightness control is fixed so that the brightness of the background image is optimal, the brightness level of the high brightness part and the brightness level of the low brightness part are expanded for the background image area. Gamma correction is performed using a gamma curve that has not been performed. Further, gamma correction is performed on the region of the presented image using a gamma curve in which the luminance level of the high luminance portion is compressed.

  When the process proceeds from step ST12 to step ST16, the control unit 51 performs the normal imaging mode notification process similarly to step ST5, and proceeds to step ST17.

  In step ST17, the control unit 51 sets the brightness control to automatic control and proceeds to step ST18. The control unit 51 automatically controls the brightness control by generating control signals CTb, CTc, and CTd according to the detection signal JB supplied from the detection unit 32 and supplying the control signals CTb, CTc, and CTd to each unit.

  In step ST18, similarly to step ST6, the control unit 51 causes the gamma correction unit 352 to perform gamma correction without dividing the presentation image and the background image, and proceeds to step ST19.

  In step ST19, the control unit 51 determines whether or not the imaging operation is finished. The control unit 51 ends the gamma correction when determining that the imaging operation is ended. If it is not determined that the imaging operation has ended, the process returns to step ST11.

  As described above, even if only the gamma curve in which the luminance level of the low luminance portion is expanded is used, it is possible to obtain a captured image in which the presentation image has no overexposure and the background image has no blackout.

  Further, in the operation shown in FIG. 14, when performing gamma correction on the region of the presentation image, gamma correction is performed using a gamma curve in which the luminance level of the high luminance portion is not compressed or the luminance level of the low luminance portion is not expanded. However, when gamma correction is performed on the region of the presentation image, a gamma curve corrected so that the presentation image can be optimized may be used. For example, a gamma curve that enhances the contrast of the presented image is used in order to optimize the presented image to be read easily. Further, a gamma curve that enhances the gradation of the high-luminance portion and the low-luminance portion may be used so that the high-luminance portion and the low-luminance portion can be easily identified in the region of the presented image.

  FIG. 15 is a flowchart showing an operation in performing gamma correction so that the presented image can be easily read.

  In step ST31, the control unit 51 causes the presentation image detection unit 31 to detect the region of the presentation image in the same manner as in step ST1, and proceeds to step ST32. In step ST32, the control unit 51 determines whether or not the condition for switching to the presentation image capturing mode is satisfied, similarly to step ST2. When determining that the condition for switching to the presentation image imaging mode is satisfied, the control unit 51 sets the imaging mode to the presentation image imaging mode and proceeds to step ST33. When determining that the imaging mode is not satisfied, the control unit 51 sets the imaging mode to normal imaging. The mode is set and the process proceeds to step ST36.

  In step ST33, as in step ST3, the control unit 51 determines that the user can identify that the imaging mode of the imaging apparatus is set to the presentation image imaging mode, and proceeds to step ST34.

  In step ST34, the control unit 51 fixes the brightness control similarly to step ST14, and proceeds to step ST35.

  In step ST35, the control unit 51 classifies the regions of the presentation image and the background image, and causes the gamma correction unit 352 of the image processing unit 35 to perform gamma correction for optimizing the images of the respective regions for each region. Proceed to In other words, the control unit 51 notifies the gamma correction unit 352 that the presentation image capturing mode has been set by the control signal CTe. The gamma correction unit 352 uses a gamma curve corrected so as to optimize the presentation image, for example, a gamma curve that enhances the contrast of the presentation image. In addition, gamma correction is performed using a gamma curve that has been corrected so as to optimize the background image, for example, a gamma curve in which the luminance level of the low-luminance portion is expanded.

  FIG. 16 shows a gamma curve that enhances the contrast of the presented image, and by using this gamma curve, gamma correction is performed to increase the gradation of medium luminance.

  When the process proceeds from step ST32 to step ST36, the control unit 51 performs the normal imaging mode notification process similarly to step ST5 and proceeds to step ST37. In step ST37, as in step ST17, the control unit 51 sets the brightness control to automatic control and proceeds to step ST38. In step ST38, similarly to step ST6, the control unit 51 causes the gamma correction unit 352 to perform gamma correction without distinguishing the presentation image and the background image, and proceeds to step ST39.

  In step ST39, the control unit 51 determines whether or not the imaging operation is finished. The control unit 51 ends the gamma correction when determining that the imaging operation is ended. If it is not determined that the imaging operation has ended, the process returns to step ST31.

  As described above, when the gamma correction of the presentation image is performed, for example, a gamma curve corrected so as to enhance the contrast is used, so that the presentation image in the captured image can be easily read. FIG. 17 shows an operation in the case of using a gamma curve corrected to enhance contrast and a gamma curve in which the luminance level of the low luminance portion is expanded. For example, the region of the presentation image is detected for the captured image shown in FIG. Here, the detected background image area is subjected to gamma correction using a gamma curve in which the luminance level of the low-luminance portion is expanded, and the detected presentation image area is enhanced in contrast. Perform gamma correction using the corrected gamma curve. Assuming that gamma correction is performed in this way, as shown in FIG. 17B, the captured image after gamma correction prevents the background image from being blacked out, and the contrast of the presented image is emphasized. A captured image with improved image quality can be obtained so that it can be easily read.

  Note that the gamma correction unit may divide the captured image into regions of the presentation image and the background image and provide a gamma correction unit for each region. The gamma correction may be performed by switching for each region.

  In addition, when performing gamma correction by switching the gamma curve for each area of the presentation image or background image with one gamma correction unit, the image signal of the presentation image and the image signal of the background image are individually extracted from the image signal of the captured image. Then, gamma correction is performed using one of the image signals. Next, the gamma curve is switched and gamma correction is performed using the other image signal, and the image signal of the presentation image after the gamma correction and the image signal of the background image are combined to form an image signal of one captured image.

  Furthermore, when gamma correction is performed by switching the gamma curve for each presentation image or background image with one gamma correction unit, whether the signal of each pixel is the presentation image or background image area in the image signal of the captured image If the gamma curve is selectively used in accordance with the image signal, the image signal of the presentation image after the gamma correction and the image signal of the background image are not combined to form an image signal of one captured image. A captured image that has been subjected to gamma correction in accordance with each of the presented image and the background image can be obtained.

It is a figure which shows the structure of an imaging device. It is a flowchart which shows a gamma correction operation | movement. It is a flowchart which shows a presentation image detection operation | movement. It is a figure which shows the acquisition process of a horizontal distribution and a vertical distribution. It is a figure which shows the detection process of the upper side and lower side of a presentation image. It is a figure which shows the detection process of the upper side and lower side of a presentation image. It is a figure which shows the detection process of the left side and right side of a presentation image. It is a figure which shows the detection process of the left side and right side of a presentation image. It is a figure for demonstrating the discrimination | determination operation | movement of whether it is a presentation image. It is a figure which shows the case where the icon display which shows having been set to the presentation image imaging mode is provided. It is a figure which shows the area | region of the presentation image in the captured image, and the area | region of a background image. It is a figure which shows the correction characteristic when performing gamma correction. It is a figure for demonstrating operation | movement when gamma correction is performed. It is a flowchart which shows operation | movement in the case of performing a gamma correction by changing a gamma curve only in the area | region of a background image. It is a flowchart which shows operation | movement in the case of performing a gamma correction so that reading of a presentation image can be performed easily. It is a figure which shows the gamma curve which emphasizes the contrast of a presentation image. The operation when using a gamma curve corrected so as to enhance the contrast and a gamma curve in which the luminance level of the low luminance part is expanded is shown. It is a figure which shows the case where the mode of a presentation is imaged with an imaging device, for example. It is a figure which shows a captured image when it images centering on a presentation image.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 ... Imaging device, 11 ... Lens part, 12 ... Diaphragm mechanism, 21 ... Imaging part, 22 ... Timing signal generation part, 23 ... A / D conversion part, 31 ... Presentation image detection unit, 32 ... detection unit, 35 ... image processing unit, 37 ... recording / reproducing unit, 38 ... display unit, 41 ... lens driving unit, 42 ... stop driving , 51... Control unit, 52... User interface unit, 211... Image sensor, 212... CDS unit, 213. 352 ... Gamma correction unit

Claims (8)

A presentation image detection unit that detects a region of a presentation image presented by the image presentation device from the captured image based on a difference in luminance or color in the captured image;
The image processing apparatus which has a gamma correction part which performs a gamma correction with a respectively different characteristic with respect to the area | region of the detected said presentation image and the area | region of the other image except the said presentation image in the said captured image.   The gamma correction unit performs gamma correction using the characteristic corrected to compress the luminance level of the high luminance part and the characteristic corrected to increase the luminance of the low luminance part as the different characteristics. Image processing apparatus.   The gamma correction unit performs gamma correction with characteristics corrected for a region of the other image when the captured image is an image whose brightness is controlled according to one of the presented image or the other image. The image processing apparatus according to claim 2, which is performed.   The gamma correction unit performs gamma correction of the region of the presentation image with a characteristic corrected so as to compress the luminance level of the high luminance part, and the other characteristic with a characteristic corrected to increase the luminance of the low luminance part. The image processing apparatus according to claim 3, wherein gamma correction is performed on an image area.   The gamma correction unit performs gamma correction with a characteristic in which contrast is enhanced with respect to a region of the presentation image when the captured image is an image in which brightness control is performed according to the presentation image. The image processing apparatus according to claim 4, which is performed.   The image processing according to claim 2, wherein the gamma correction unit performs gamma correction by switching correction characteristics according to whether the supplied image signal is a region of the presentation image or another region excluding the presentation image. apparatus. A presentation image detection step of detecting a region of a presentation image presented by the image presentation device from the captured image based on a difference in luminance or color in the captured image;
An image processing method comprising: a gamma correction step of performing gamma correction with different characteristics for each of the detected presentation image area and the other image area excluding the presentation image in the captured image. An imaging unit that images a subject and obtains a captured image;
A presentation image detection unit that detects a region of a presentation image presented by an image presentation device from the captured image based on a difference in luminance or color in the captured image;
An imaging apparatus comprising: a gamma correction unit configured to perform gamma correction with different characteristics for each of the detected presentation image area and another image area excluding the presentation image in the captured image.