JP5046809B2 - Image processing apparatus and method, and imaging apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To make a correction so that a desired object may be displayed in desired color. <P>SOLUTION: An image processing device for processing an image signal of an image obtained by picking up an image has: a storage means 108 for storing color data of a specific object in advance; a detecting means 126 for detecting a preset object from the image; a determination means 106 for determining whether the object detected by the detecting means is the specific object stored in the storage means; and an image processing means 106 for converting color data in an image area representing the detected object on the basis of the specific object stored in the storage means when it is determined by the determination means that the object is the specific object stored by the storage means. <P>COPYRIGHT: (C)2009,JPO&amp;INPIT

Description

  The present invention relates to an image processing apparatus and method, and an imaging apparatus, and more particularly to an image processing apparatus and method and an imaging apparatus that can detect an image area of an arbitrary object from an image obtained by imaging.

  Conventionally, there is an apparatus having a function of repeatedly detecting an object that satisfies a predetermined condition from image data. In an imaging apparatus having such an apparatus, many techniques for differentiating an image processing method between an object part that satisfies a predetermined condition and other parts of an image signal obtained by imaging are disclosed. (For example, refer to Patent Document 1). Furthermore, a technique is disclosed in which image data is processed by a unique image processing method based on the characteristics of a detected object when an object that satisfies a predetermined condition is detected (see, for example, Patent Document 2). ).

JP 2006-109310 A JP 2005-128600 A

  However, as in Patent Document 2, in an imaging apparatus that detects an object that satisfies a predetermined condition and performs a unique image processing method that makes use of the characteristics of the detected object, the image processing result desired by the photographer is not necessarily obtained. Was not always possible.

The present invention has been made in view of the above problems, and an image processing apparatus of the present invention that processes an image signal of an image obtained by performing imaging stores a plurality of color regions corresponding to a specific object in advance. Storage means, detection means for detecting an object having a preset characteristic from the image, and whether the object detected by the detection means is the specific object stored in the storage means A determination means for determining whether or not an image area representing the object determined to be the specific object in response to the determination by the determination means being determined to be the specific object stored in the storage means; An image area representing the detected object in a coordinate system representing the color data and the color data among a plurality of color areas corresponding to the specific object stored in the storage means. Color day And an image processing means for performing image processing for converting the color data in the color area in a shortest distance from.

  The image pickup apparatus of the present invention includes an image pickup unit that converts a subject optical image into an electrical image signal and outputs the image signal, and the image processing apparatus.

Further, the image processing method of the present invention for processing an image signal of an image obtained by performing imaging includes a detection step in which the detection means detects an object having a preset feature from the image, and a determination means. However, the determination step for determining whether the object detected by the detection step is a specific object stored in the storage means, and the image processing means are stored in the storage means by the determination step. A plurality of color data corresponding to the specific object pre-stored in the storage means are stored in the storage means for color data of an image area representing the object determined to be the specific object in response to being determined to be the specific object In the coordinate system representing the plurality of color regions and the color data, the color data of the image region is converted from the color data of the image region representing the detected object into color data in the color region at the shortest distance. Image processing And an image processing step of performing.

  ADVANTAGE OF THE INVENTION According to this invention, it can correct | amend so that a desired target object may be displayed with a desired color.

  The best mode for carrying out the present invention will be described below in detail with reference to the accompanying drawings.

  A schematic configuration of the imaging apparatus 1 in the present embodiment will be described with reference to FIG.

  Reference numeral 101 denotes a photographing lens including a focus lens, and 102 denotes a shutter having an aperture function. Reference numeral 103 denotes an image pickup device represented by a CCD or CMOS sensor that converts a subject optical image into an electrical image signal as an example of an image pickup means. Reference numeral 104 denotes an analog output output from the image pickup device 103 as a digital signal (digital image data). A / D converter for converting to

  A timing generation circuit 105 supplies a clock signal and a control signal to the image sensor 103, the A / D converter 104, and the D / A converter 110, and is controlled by the memory control circuit 109 and the system control unit 107.

  An image processing circuit 106 performs predetermined pixel interpolation processing and color conversion processing on the image data from the A / D converter 104 or the image data from the memory control circuit 109. Further, the image processing circuit 106 includes a face detection unit 126, and detects a face from image data and acquires information of the detected face. The face information includes the presence / absence of a face, reliability, which is an index of the probability of being a face when detected, the position and size of the face. The face detection unit 126 is an example of a detection unit, and has a configuration in which a human face is used as a detection target.

  As a face detection technique, for example, there are a method using learning typified by a neural network, and a method of detecting a part having physical shape features such as eyes and nose from an image region by template matching. In addition, there is a method of detecting image feature quantities such as skin color and eye shape and using statistical analysis (see, for example, Japanese Patent Laid-Open Nos. 10-232934 and 2000-48184). .

  Further, the image processing circuit 106 performs a predetermined calculation process using the captured image data. Based on the obtained calculation result, the system control unit 107 performs AF (autofocus) processing as focusing control, and AE (automatic exposure) processing and EF (flash pre-flash) processing as exposure control. Further, AWB (auto white balance) processing is also performed.

  The digital image data output from the A / D converter 104 is sent to the image memory 108 via the image processing circuit 106 and the memory control circuit 109, or directly from the A / D converter 104 via the memory control circuit 109. It is written in the image display memory 111.

  Reference numeral 108 denotes an image memory for storing captured still images and moving images, and has a sufficient storage capacity for storing a predetermined number of still images and moving images for a predetermined time. Thereby, even in the case of continuous shooting or panoramic shooting in which a plurality of still images are continuously shot, it is possible to write a large amount of images to the image memory 108 at high speed. The image memory 108 can also be used as a work area for the system control unit 107. Furthermore, in the present embodiment, the image memory 108 is also used to hold a specific person and information on the face color of the person.

  Reference numeral 111 denotes an image display memory. Display image data written in the image display memory 111 is displayed on the display unit 112 via the D / A converter 110. The display unit 112 can arbitrarily turn on / off the display according to an instruction from the system control unit 107. When the display is turned off, the power consumption of the imaging apparatus 1 can be significantly reduced.

  An operation unit 113 includes various switches represented by a shutter button, operation members such as a mode dial and a slider, an operation member for instructing a zoom magnification, a soft switch using the display unit 112, and the like. A user instruction input using the operation unit 113 is sent to the system control unit 107. In this embodiment, the shutter button is a known two-stage shutter button, and a recording preparation instruction is input when the shutter switch SW1 is turned on by a first stroke (for example, half-pressing) of the shutter button. In response to the input of the recording preparation instruction, operations such as AF processing, AE processing, AWB processing, and EF processing are started. Further, the shutter switch SW2 is turned on by a second stroke (for example, full press) of the shutter button, and an instruction to start a series of processing including exposure processing, development processing, and recording processing is given. Further, the operation unit 113 can set a face detection mode for detecting a face from a captured image.

  Next, the configuration and processing of the image processing circuit 106 will be described in more detail with reference to FIGS.

  FIG. 2 is a block diagram showing the configuration of the image processing circuit 106, and FIGS. 3 and 4 show the wavelengths (colors) at which each pixel of the image sensor 103 mainly performs photoelectric conversion. FIG. 3 shows a case where a primary color signal is output, and FIG. 4 shows a case where a complementary color signal is output. In general, since each color filter is arranged in each pixel, FIG. 3 is referred to as a primary color filter array and FIG. 4 is referred to as a complementary color filter array.

  The image sensor 103 that outputs primary color signals as shown in FIG. 3 is divided into blocks with R, G1, G2, and B as units. The R, G1, G2, and B signal values are read from the image sensor 103 and sent to the white balance processing unit 301. The white ballast coefficient and the color temperature of the light source are such that white in the image becomes a signal representing white. Desired. Thereafter, the color evaluation values for determining the color temperature of the light source are Cx and Cy (see the following formula (1)), and the white balance coefficients of the color signals R, G1, G2, and B are kWB_R, kWB_G1, kWB_G2, and kWB_B, respectively. And

  Cx and Cy are calculated for each block. The calculation formula (1) for the Cx and Cy values is shown below.

  The calculation method of the white balance coefficients kWB_R, kWB_G1, kWB_G2, and kWB_B is shown below. As shown in FIG. 5, the white axis 201 is set by photographing white in advance from a high color temperature to a low color temperature and plotting the respective color evaluation values Cx and Cy. Since white varies slightly with an actual light source, the region 202 indicating white is given a width. This range is called a white detection range. At this time, when the Cx and Cy values calculated for each block of the image sensor 103 having the primary color filter shown in FIG. 3 are included in the white detection range 202, it is assumed that the block is white. Further, the integral values Sum_R, Sum_G1, Sum_G2, and Sum_B of the pixels included in the white detection range 202 are calculated, and the white balance coefficient is calculated by the following equation (2).

  In response to the above calculation results, the white balance processing unit 301 applies the white balance coefficient to the digital image signal output from the image sensor 103 via the A / D converter 104, so that the white in the image is white. The white balance is corrected so that the signal becomes a representative signal. The white balance corrected image signal is sent to the luminance signal creation / selection unit 308 and the interpolation processing unit 302. The interpolation processing unit 302 uses the pixels at positions A (R), B (G1), C (G2), and D (B) from the pixel array of the image sensor 103, respectively, and performs A, B, C, Create surface data of D signal.

  The matrix calculation processing unit 303 performs color conversion for each pixel using Expression (3).

In the above formula (3), M11 to M43 are coefficients.
The digital image signal that has been subjected to matrix calculation processing is gained by the color difference gain calculation processing unit 304. Here, the Rm, Gm, and Bm signals are converted into Y, Cr, and Cb signals according to Equation (4), and further gains are applied to the Cr and Cb signals according to Equation (5). ) Is converted into Rg, Gg, and Bg signals.

Cr' = G1 × Cr
Cb' = G1 × Cb …（５）

Cr '= G1 × Cr
Cb '= G1 x Cb (5)

The digital image signal subjected to the color difference gain calculation process is sent to the gamma processing unit 305. The gamma processing unit 305 performs data conversion using the following equations (7) to (9). GammaTable is a one-dimensional lookup table.
Rt = GammaTable [Rg] (7)
Gt = GammaTable [Gg] (8)
Bt = GammaTable [Bg] (9)
The gamma-processed digital image signal is sent to the hue correction calculation processing unit 306. The Rt, Gt, and Bt signals are converted into Y, Cr, and Cb signals by the arithmetic expression (10), and the Cr and Cb signals are corrected by the expression (11), and the expression (12) (the inverse matrix of the expression (11)) Is converted into Rh, Gh, and Bh signals.

However, H11 to H22 are coefficients

The digital image signal subjected to the hue correction calculation process is sent to the color difference signal conversion processing unit 307.
In the color difference signal conversion processing unit 307, a UV signal is created using Expression (13).

  On the other hand, the signal sent to the luminance signal creation / selection unit 308 is processed by a processing method selected depending on whether a primary color filter or a complementary color filter is used for the image sensor 103. If the image sensor 103 uses the primary color filter of FIG. 3, a luminance signal is generated by the primary color luminance signal creation processing unit 309, and if the image sensor 103 uses a complementary color filter, a complementary color luminance signal creation process. A luminance signal is created by the unit 310. In the case of the imaging apparatus 1 shown in FIG. 1, it is known whether a primary color filter or a complementary color filter is used in the manufacturing stage. Therefore, the luminance signal creation selection unit 308 may be omitted, and either the primary color luminance signal creation processing unit 309 or the complementary color luminance signal creation processing unit 310 may be configured.

  In the following, how to create a primary color luminance signal and a complementary color luminance signal will be described.

  In the case of the primary color filter as shown in FIG. 3, all the R and B signals are set to 0, and the two-dimensional filter processing having the coefficients shown in FIG. In the case of a complementary color filter as shown in FIG. 4, a C, M, Y, G signal subjected to a two-dimensional filter process having the coefficients shown in FIG.

  The luminance signal generated by the primary color luminance signal generation processing unit 309 or the complementary color luminance signal generation processing unit 310 is subjected to edge enhancement processing by the high frequency enhancement processing unit 311 and further subjected to gamma conversion processing by the gamma processing unit 312, and the Y signal is converted. Created.

<First Embodiment>
Next, the face color correction processing operation in the imaging mode of the imaging apparatus 1 having the above-described configuration according to the first embodiment of the present invention will be described with reference to the flowchart of FIG.

  In the first embodiment, what color the desired object (specific object) is to be expressed in is stored (registered) in the image memory 108 in advance, and the object registered in the photographed image. Is corrected so that the color of the object is expressed by the registered color. FIG. 8 shows an example of the format of the registered object and color data to be expressed. FIG. 8 shows a case where the object is the face of a specific individual. Note that FIG. 8 is an example, and the format is not limited to this.

  After turning on the battery by turning on the battery in step S101, the system control unit 107 confirms the set mode among various modes in which the imaging apparatus 1 can operate (step S102), and whether or not the imaging mode is set. Is determined (step S103). Since processing described below is not performed in modes other than the imaging mode, in the first embodiment, the process returns to step S102 and the confirmation of the set mode and the determination of whether the imaging mode is set are repeated.

  When the imaging mode is set, if the display unit 112 is OFF, it is turned ON (step S104), and live image display (EVF display) is started (step S105). Note that live images are displayed by reading frame images from the image sensor 103 at predetermined intervals and sequentially displaying them on the display unit 112.

  Next, the system control unit 107 determines whether or not the face detection mode is set by the operation unit 113 (step S106). When the face detection mode is set, the face detection unit 126 periodically checks whether or not a face is present in a frame image captured at a predetermined cycle for displaying a live image (step S107). The face detection need not be performed on all the frame images taken for the live image, and may be performed at an arbitrarily set cycle. Each time the frame image is examined, if a face is detected, including the presence or absence of a face, information about the detected face is output, stored, and updated as needed (step S108). At this time, a frame indicating the detected face area may be displayed.

  If the face detection mode is not set in step S106, or if it is determined in step S107 that no face exists, the process proceeds to step S115. When the shutter button included in the operation unit 113 is pressed and the shutter switch SW1 is turned on, normal image processing is executed (step S116), and the processed image is previewed on the display unit 112. If the preview-displayed image is an image desired by the user, the shutter button SW2 is further pressed to turn on the shutter switch SW2 (YES in step S117), and imaging is performed (step S119).

  On the other hand, if the shutter switch SW1 is not turned on in step S115, or after the preview in step S116, the shutter switch SW2 is not turned on and SW1 is also turned off, the process returns to step S106.

  When a face is detected (YES in step S107) and a frame indicating the face area is displayed (step S108), the state of the shutter switch SW1 is determined (step S109). If OFF, the process returns to step S106 and is turned ON. Then, it progresses to step S110. In step S110, for example, it is determined whether or not the face detected from the live image (EVF display) is a registered person from the information related to the individual face registered in the image memory 108 as shown in FIG. . If it is determined that the person is not registered, default image processing suitable for the face of the person is executed (step S114). On the other hand, if it is determined that the person is a registered person, face color data is acquired from information about the face of the person (step S111), and the face color of the detected face is corrected based on the data after shooting. .

  Here, the method for correcting the face color performed in step S111 will be described.

  FIG. 9 shows an example in which facial color data and facial color development allowable areas registered in the image memory 108 are expressed in the L * a * b * color system. Here, a region from the point indicating the facial color of the registered person to a region separated by a predetermined color difference ΔE is represented as a facial color development allowable region. In step S111, first, the default image processing used in step S114 is executed on the image data of the detected face image area. Note that the digital image signal can be converted into the L * a * b * color system from the Rh, Gh, and Bh signals obtained by the above equation (20) as follows.

In the above formula (15), N1 to N9 are constants. And let Xn, Yn, Zn be the tristimulus values of the illumination light,

It becomes. In this way, the digital image signal of the L * a * b * color system is converted.
If the obtained color data is included in the facial color development allowable region in FIG. 9, the obtained color data is used as it is and a preview is displayed in step S112.

  On the other hand, as a result of performing the default image processing, when the obtained color data is not included in the facial color development allowable region, the processing described below with reference to FIG. 10 is performed.

  FIG. 10 shows registered face color data, a face color coloration allowable area, and face color data (face color data obtained by simulation) obtained as a result of executing default image processing. In this case, the shortest distance between the image data obtained as a result of image processing and the facial color development allowable area is obtained, and the color data of the facial color development allowable area at the shortest distance (within a preset range from the registered facial color data) A certain color data).

  Further, it may be structured such that ideal skin color data is acquired in advance by a color capturing means or the like and is registered in the format of FIG. 8 to acquire the data as a face color.

  Returning to the flowchart of FIG. 7, the result corrected in step S111 is displayed as a preview (step S112). An example of the display method here is shown in FIG. In the example shown in FIG. 11, the face area is enlarged and displayed at the lower right of the screen (or a position that does not interfere with visual recognition of the image) that represents the face with the color data acquired in step S111. is doing. Of course, the display method is not limited to the example shown in FIG. 11. For example, only the color indicated by the color data may be displayed in a patch format or numerically.

  The user confirms this preview display, and determines whether or not the color development is acceptable by the operation unit 113 or a method of selecting an imaging mode (step S113). If it is determined that the user is not allowed, normal face priority image processing is executed, and the processed image is previewed on the display unit 112 (step S114). On the other hand, if the image using the color data acquired in step S111 is an image desired by the user, the shutter switch SW2 is turned on by further pressing the shutter button (YES in step S117), and imaging is performed (step S117). Step S119). The image data obtained by imaging in step S119 is processed so as to have a face color corrected by the skin color data used for the preview display in step S112 or S114.

  On the other hand, if the shutter switch SW1 is not turned on in step S109, or after the preview in step S112 or S114, if the shutter switch SW2 is not turned on and SW1 is also turned off, the process returns to step S106.

  As described above, according to the first embodiment, correction can be performed so that a desired object is displayed in a desired color.

<Second Embodiment>
Next, a second embodiment of the present invention will be described.

  FIG. 12 is a flowchart showing the operation of the image pickup apparatus 1 in the image pickup mode according to the second embodiment of the present invention. In the second embodiment, a case where a plurality of faces are detected during preview will be described. In addition, the same reference number is attached | subjected to the process similar to the said 1st Embodiment, and only a different part is demonstrated.

  When the switch SW1 is turned on in step S109, it is determined whether or not all faces detected on the live view (EVF display) in step S210 are registered persons. If it is determined that all faces are unregistered, face-priority image processing that is commonly performed for all persons is executed (step S114).

  On the other hand, if it is determined in step S211 that all the faces detected on the live view (EVF display) are any registered persons, the process of step S111 described above is executed for all the faces. Get color data. Each face is preview-displayed in the color indicated by the obtained color data (step S216).

  If it is found on the EVF display that a person who has registered personal information and an unregistered person are mixed (NO in step S211), the registered person is corrected based on the personal face color data. This is performed (step S212). The unregistered person is corrected so as not to be unnatural based on the correction amount executed for the registered person (step S213). After correcting each face color, the correction result is displayed as a preview in step S214.

  The subsequent processing is the same as the processing described in the first embodiment with reference to FIG.

  As described above, according to the second embodiment, even when a plurality of objects are detected, each of the objects can be corrected so as to be displayed in a desired color.

<Other embodiments>
Note that the present invention can be applied to a system composed of a plurality of devices (for example, a host computer, an interface device, a camera head, etc.), or a device (for example, a digital still camera, a digital video camera, etc.) composed of a single device. You may apply to.

  The object of the present invention can also be achieved as follows. First, a storage medium (or recording medium) that records a program code of software that implements the functions of the above-described embodiments is supplied to a system or apparatus. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also the following can be achieved. That is, when the operating system (OS) running on the computer performs part or all of the actual processing based on the instruction of the read program code, the functions of the above-described embodiments are realized by the processing. It is. Examples of the storage medium for storing the program code include a flexible disk, hard disk, ROM, RAM, magnetic tape, nonvolatile memory card, CD-ROM, CD-R, DVD, optical disk, magneto-optical disk, MO, and the like. Can be considered. Also, a computer network such as a LAN (Local Area Network) or a WAN (Wide Area Network) can be used to supply the program code.

1 is a block diagram illustrating a schematic configuration of an imaging apparatus according to an embodiment of the present invention. It is a block diagram which shows the structural example of the image processing circuit in embodiment of this invention. It is a figure which shows an example of a primary color filter arrangement | sequence. It is a figure which shows an example of a complementary color filter arrangement | sequence. It is a figure which shows a white detection range. It is a figure which shows the filter processing coefficient for a luminance signal calculation. It is a flowchart which shows the operation | movement of the face color correction process at the time of the imaging mode in the 1st Embodiment of this invention. It is a figure which shows the format of the information of the registered target object and the color to express. FIG. 9 is a diagram showing the registered color data in the L * a * b * color system. It is a figure which shows the concept of the correction method of a face color. It is a figure which shows the example of a preview display in the 1st Embodiment of this invention. It is a flowchart which shows the operation | movement of the face color correction process at the time of the imaging mode in the 2nd Embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Imaging device 101 Shooting lens 102 Shutter 103 Image pick-up element 104 A / D converter 105 Timing generation circuit 106 Image processing circuit 107 System control part 108 Image memory 109 Memory control circuit 110 D / A converter 111 Memory 112 Display part 113 Operation part 126 Face detection unit

Claims (13)

An image processing apparatus that processes an image signal of an image obtained by imaging,
Storage means for storing in advance a plurality of color regions corresponding to the specific object;
Detecting means for detecting an object having a preset characteristic from the image;
Determining means for determining whether the object detected by the detecting means is the specific object stored in the storage means;
In response to determining that the specific object is stored in the storage means by the determining means, color data of an image area representing the object determined to be the specific object is stored in the storage means. Among the plurality of color areas corresponding to the stored specific object, in the coordinate system representing the plurality of color areas and the color data, the shortest distance from the color data of the image area representing the detected object An image processing apparatus comprising: image processing means for performing image processing for conversion into color data in a certain color area. The storage means stores a combination of a plurality of specific objects and color regions corresponding to the plurality of specific objects,
When a plurality of preset objects are detected by the detection means, the determination means is that each of the detected plurality of objects is one of the specific objects stored in the storage means. To determine whether
The image processing means stores color data of an image area representing the detected object determined to be any one of the specific objects stored in the storage means, stored in the storage means, The image processing apparatus according to claim 1, wherein the image processing is performed based on a color region corresponding to the specific object corresponding to the detected object. The image processing means performs the image processing based on a color difference between a color area corresponding to a specific object stored in the storage means and color data of an image area representing the detected object. The image processing apparatus according to claim 1 or 2.   The image processing means, when there is an object detected by the detection means determined to be the specific object, the detected object determined not to be the specific object by the determination means 4. The image processing apparatus according to claim 1, wherein image processing is performed on color data of an image area to be represented based on image processing executed on the specific target object. 5.   5. The image processing apparatus according to claim 1, further comprising a display unit that displays an image processed by the image processing unit before the image signal is recorded. 6.   6. The image processing apparatus according to claim 5, wherein the detected object processed by the image processing means is enlarged and displayed on the display means.   6. The image processing apparatus according to claim 5, wherein color data of an image area representing the detected object processed by the image processing unit is displayed on the display unit in a patch format.   6. The image processing apparatus according to claim 5, wherein color data of an image area representing the detected object processed by the image processing unit is displayed as a numerical value on the display unit.   The image processing apparatus according to claim 1, wherein the object is a human face. Imaging means for converting a subject optical image into an electrical image signal and outputting it;
An imaging apparatus comprising: the image processing apparatus according to claim 1. An image processing method for processing an image signal of an image obtained by imaging,
A detecting step for detecting an object having a preset feature from the image;
A determination step of determining whether or not the object detected by the detection step is a specific object stored in the storage unit;
In response to the determination by the determination step that the image processing means is the specific object stored in the storage means, color data of an image area representing the object determined to be the specific object is obtained. The color of the image area representing the detected object in the coordinate system representing the color area and the color data among the color areas corresponding to the specific object stored in advance in the storage means And an image processing step of performing image processing for converting the color data into the color data in the color region at the shortest distance from the data.   The program for making a computer perform each process of the image processing method of Claim 11.   A computer-readable storage medium storing the program according to claim 12.

Images