JP4822690B2 - Image processing method and apparatus, and print service system - Google Patents

Description

  The present invention relates to an image processing method and apparatus therefor, and a print service system, and more particularly to a color image device such as a digital camera, a display, a scanner, and a printer, and a color image correction process in image processing software of the color image device. is there.

  Conventionally, various image processing such as adjustment of saturation, color tone, contrast, and gradation has been performed on digital images. Conventionally, when performing such image processing, an operator who has specialized knowledge about the image uses specialized or dedicated software and makes full use of empirical knowledge so that the image processing can be performed on a computer monitor screen. In general, trial and error are performed while confirming the result to obtain a suitable image.

  In recent years, the popularity of digital cameras has been increasing, driven by the popularity of the Internet. This is because the result (data) obtained by photographing with a digital camera is data in a file format that can be easily read by a computer. For example, it is possible to easily store an image photographed by a digital camera on a WWW (World Wide Web) server and open it to a third party. With the spread of such digital cameras, it is also true that digital image data has increased around users who have never had much connection with images.

  However, users of digital cameras are not users who have sufficient knowledge about cameras that can freely use conventional analog cameras, but rather users who are familiar with computers but do not have enough knowledge about cameras. It is done. Therefore, images taken with a digital camera are not necessarily only images taken under suitable conditions. That said, even if an image was taken under unfavorable conditions, it may not always be discarded if the content is important to the photographer. For this reason, there is a demand for an image correction method that can obtain an appropriate image from this image even if the image is taken under unfavorable conditions.

  For example, the color matching technique corrects an image so as not to be affected by the characteristics of input / output devices, and requires color correction faithful to the actual product. On the other hand, when dealing with a photographic image taken with a digital camera, it is also important to perform image correction so that a person feels better than being true to the real thing. In the case of portraits, the main area of interest is the face of the person who is the subject, and it is preferable that the face area has the appropriate brightness (skin color). In such a case, it can be said that it is preferable that the correction is made so that the brightness (skin color) is adjusted.

Therefore, as in Patent Document 1, when the correction method performed in the past by the user is recorded together with the image feature and a new image is corrected, the database is searched with the new image feature and the corresponding correction is performed. By applying the method, image correction considering personal preference was performed.
JP 2003-187215 A

  However, the method according to Patent Document 1 reflects the preference of the operator of the image processing apparatus, and does not reflect the preference of the subject of the image. Therefore, it cannot be used for print services.

  In addition, when image correction has not been performed in the past, general correction is performed with reference to a common correction database, so that different preferences cannot be reflected depending on the person.

  In view of the above-described conventional problems, the present invention provides an image processing method and apparatus for performing image correction reflecting the preference of the user (the subject of the image), and a print service system.

In order to achieve the above object, an image processing apparatus according to the present invention includes an image input unit that inputs image information, and a region setting that sets a face region that includes a human face in the image information input from the image input unit. means, for each person, and face information representing a feature of the face of the person thereof, and profile information storage means for storing profile information and an image correction information including the target color corresponding to the image of the face of the person thereof, wherein A collation unit for collating face information in the profile information with respect to the face area set by the area setting unit, a target color in the profile information including face information matched by the collation by the collation unit, and a representative of the face area and having an image correcting means for performing image correction on the facial region by the smaller amount of color correction is large difference between the colors.

Further, the print service system of the present invention is installed in a remote place with respect to a plurality of user terminals and the plurality of user terminals, functions as a server, and can communicate with the plurality of users via a network. The image processing apparatus according to any one of 1 to 5 , wherein a desired image data transmitted by a user via the network is corrected in accordance with profile information transmitted by the user and transmitted to a print shop. And a print shop for receiving and printing the corrected image data in accordance with a print instruction from the image processing apparatus.

  Furthermore, according to the present invention, there are provided an image processing method corresponding to the image processing of the image processing apparatus, a program for realizing the image processing method by a computer, and a recording medium storing the program.

  As described above, according to the present invention, there is an effect that the image correction reflecting the user's preference can be performed.

  Hereinafter, the configuration and operation of an image processing apparatus according to an embodiment of the present invention will be described in detail with reference to the accompanying drawings.

<Conceptual Configuration Example of Image Processing Apparatus of Present Embodiment>
FIG. 1 is a diagram illustrating a conceptual configuration of an image processing apparatus according to the present embodiment.

  In FIG. 1, reference numeral 101 denotes an image input unit for inputting image information. The image input unit 101 of this example includes input from an image input device such as a digital camera or a scanner, an image storage medium such as a floppy (registered trademark) disk, CD, or memory card, and reception via a network.

  Reference numeral 102 denotes an area setting unit that detects whether or not a preset target is included in the image information input from the image input unit 101 and sets the detected area. In this example, a human face is set as a preset target. Whether or not the face is a person is detected based on, for example, the position of a contour or a component (such as eyes and mouth).

  Reference numeral 103 denotes a collation unit that collates a partial image of an area set by the area setting unit 102 with profile information stored in a profile information storage unit 106 to be described later, and determines an image correction method based on the matching profile information ID. To get.

  An image correction unit 104 performs image correction on the face area set by the area setting unit 102 using the correction method acquired by the matching unit 103.

  An image output unit 105 outputs the corrected image. For example, the image output unit 107 includes output to an output device such as a display or a printer, output to an image storage medium such as a floppy (registered trademark) disk, CD, or memory card, and further transmission via a network.

  Reference numeral 106 denotes a profile information storage unit that stores profile information. The profile information storage unit 106 will be further described with reference to FIGS.

<Specific Configuration Example of Image Processing Apparatus of Present Embodiment>
FIG. 2 is a block diagram illustrating a specific configuration example of the image processing apparatus according to the present embodiment.

  The image processing apparatus stores an arithmetic processing CPU 201 that executes instructions according to programs stored in the ROM 202 and the RAM 203, a ROM 202 that stores programs and data necessary for the control according to the present embodiment and other controls, and temporary data. Input images from devices such as RAM 203, drive I / F 204 that realizes an interface with an external storage device such as IDE or SCSI, HDD 205 that stores moving image or feature amount or program for moving image search, digital camera or scanner, etc. An image input unit 206, a buffer memory 207 for storing temporary data, an input unit 208 for inputting from an operator such as a keyboard and a mouse, a display unit 209 such as a cathode ray tube or a liquid crystal display, and a network such as the Internet or an intranet. A network I / F 210 such as a modem or a LAN that performs a connection, a printer I / F 211 that realizes an interface with a printer, a printer 212 that prints and outputs an image, a USB interface 216 that reads and writes profile information, and a storage medium 217 that stores profile information The data is input / output to / from each other via a bus 213 connecting them.

  2 shows a configuration in which the profile information is read from the storage medium 217 that stores the profile information. However, the profile information is image data from other input means, for example, the image input unit 206, or the network interface 210. It may be input via communication.

  In this image processing apparatus, WINDOWS (registered trademark) XP of Microsoft Corporation is installed as an operating system, and the image processing apparatus is installed as an application operating on the WINDOWS (registered trademark) XP.

  The RAM 203 of the present embodiment has a data storage area that is temporarily used during the arithmetic processing of the CPU 201 and a program load area for loading the application program of the present embodiment executed by the CPU 201 from the HDD 205 or the like.

  The data storage area includes an input image data area 203a for storing input image data, an image data area 203b for each detected face area i detected as a face area, a calculation likelihood area 203c, and a profile ID area 203d for a matching result. A correction method information area 203e, a correction data area 203f, a likelihood threshold area 203g, and an output image data area 203h.

  The program load area includes an image processing program 203i (FIG. 5), an image input module 203j, a face area detection module 203k (FIG. 6), a profile discrimination module 203m (see FIG. 8), and an image correction module 203n (FIG. 9). The image output module 203p is loaded and executed by the CPU 201.

<Example of profile information>
An example of an overview of the storage medium 217 for storing profile information is shown in FIG.

  It can be inserted into a general-purpose USB terminal, and profile information is stored in an internal nonvolatile memory.

  A configuration example of the profile information 40 stored in the profile information storage medium is shown in FIG. The contents of each information will be described with reference to this figure.

  The profile ID 41 is an identifier that uniquely identifies profile information. Furthermore, in order to prevent others from illegally using the ID, for example, unauthorized use prevention information obtained by performing a predetermined calculation from an ID such as a checksum may be added to the ID. In order to generate this ID, for example, a predetermined calculation is performed from information such as a MAC address for identifying the terminal and the creation time, and further, unauthorized use prevention information is added.

  The collation information 42 is for identifying an individual from the face image to be corrected, and the information stored differs depending on the previous collation information type 421. The collation information type 421 indicates the data structure type of the subsequent type-specific collation information 422, and is used for handling different types of collation information. When matching with the image to be corrected, the detection performance is good and bad, and an appropriate price is set from the equipment required for photographing, and the owner can select it. In the present embodiment, “face image 422-1”, “stereoscopic information 422-2”, and “face feature amount” are defined as examples of the collation information type.

  When the collation information type is a face image (422-1), the number of images 422a and the number of face images 422b corresponding to the number of images are stored. In the present embodiment, one image is binary data that is 160 pixels wide and 240 pixels high and is irreversibly encoded by JPEG. This may be a face image taken by the owner himself with a digital camera or the like. Of course, an image cut out from the face area on the image to be corrected may be used.

  The feature amount in the case of collation is generated from the face image at an arbitrary timing between the creation of the profile and the time of image correction. A conventional technique can be used as the feature amount extraction method, and is not limited. For example, M.M. Turk and A.M. Pentland: Eigenfaces for recognition. Journal of Cognitive Neuroscience, Vol. 3, No. 1, PP. 71-86, March 1991). In this embodiment, a face image is represented by a region of horizontal 48 pixels and vertical 64 pixels, and the luminance value of each pixel is projected onto a partial space specified by a base vector obtained by performing principal component analysis to be a feature amount. .

  When the collation information type is stereoscopic information (422-2), the face is scanned with a 3D scanner, and surface information such as the number of poses 422c, shape information (vertex information 422d), and texture (texture information 422e) is stored. The format may be a data format specific to the application other than the DXF format. It is possible to scan with a plurality of poses with different facial expressions, and to store three-dimensional information of a plurality of poses.

  When the collation information type is a face feature amount (423), the face feature amount is generated from the face image at an arbitrary timing between the creation of the profile and the time of image correction, as in the case of the face image. In this embodiment, since face discrimination is performed from a still image, feature amounts are obtained from data that is imaged (rendered) after setting the face orientation and light source in various combinations from the three-dimensional information. When the collation information type is a feature amount, only the feature amount obtained by the above-described method is recorded from face images taken in various poses. Since the memory required for recording is small, the cost is reduced, but there is a demerit that the feature cannot be recreated in the future.

  Other than the above, any feature that records facial features of the face, such as face image information with depth information using a laser distance meter using laser reflection, etc., is within the scope of the present invention.

  The discrimination information 43 is information for discriminating whether the user is the owner of the profile from the collation result with the feature amount of the profile. It consists of a basis vector for discriminant analysis, a conversion table from distance between feature amounts to likelihood, and a threshold value.

  The basis vector 431 for discriminant analysis prepares training feature data divided into two classes, whether or not the person is, so that the ratio of the variance of the feature quantity within the class and the variance of the feature quantity between classes is maximized. It is obtained by searching for a subspace. The more training data, the better the discrimination performance, so it is possible to further improve accuracy by adding the face image identified when performing image correction to the image of the collation information and performing the discrimination analysis again. is there. In the case of 3D information, personal identification performance can be improved because it can be generated abundantly by rendering (rendering) after setting the face orientation and light source in various combinations compared to the face image. .

  The inter-feature amount distance to likelihood conversion table 432 is a table for converting the Euclidean distance of the feature amount into normalized likelihoods from 0 to 1. In general, the distribution of the distance between feature amounts obtained from many face images collected is obtained by the following formula.

However, h (i) is the occurrence frequency of the distance i. The accuracy of discrimination can be improved by collecting the faces detected from the past images to be corrected and obtaining the distance distribution between the feature amounts.
Further, 0.95 is set as an initial value as the likelihood threshold 433, but a value desired by the owner of the profile can be set.

  The expiration date 44 is information indicating the validity period of this profile. As a result, the influence of aging on the face can be reduced, and long-term sales can be secured by charging at the time of shooting.

  The correction method 45 is information indicating a method for correcting an image when the profile information matches.

  The correction method 45 includes a flesh color target luminance 451, a flesh color target saturation 452, a flesh color target hue 453, a flesh color soft focus strength 454, etc., and the owner can select a preferred correction method. Is. It is not preferable to present the correction items and correction contents as they are for the general public. About 10 types of typical correction parameter combinations may be prepared for each gender and age group, and selected from them.

  The profile information can be generated by going to a shop for shooting or by operating a predetermined application for setting the profile information on a client terminal. Since the profile information is digital data to which a profile ID is assigned, it is easy to link the profile ID and information about an owner (not shown). Needless to say, one owner can have a plurality of profiles, and the profile information of a plurality of persons can be stored in one storage medium. The profile information can be stored in a medium similar to a storage medium that can store the program. In the case of a system connected to a network, it may be recorded in a distributed state regardless of whether it is a server or a client.

<Example of Operation of Image Processing Apparatus of Present Embodiment>
Next, an outline of the image correction processing in the image processing apparatus having such a configuration will be described with reference to the flowchart of FIG.

  After inputting the image data, a face area is detected from the image to be corrected in step S301. Processing branches at step S302 depending on whether an area has been detected. If an area is detected, the process proceeds to step S303, and if not, the process ends.

  In step S303, a face area is determined for each detected face area. Here, each face area is collated with the profile collation information, and the confirmed profile ID is output. Processing branches at step S304 depending on whether a profile ID has been output. If the profile ID is output, the process proceeds to step S305, and if not, the process ends.

  In step S305, an image correction method corresponding to the profile ID is acquired, and in step S306, the face area of the input image is corrected.

  Naturally, correction to compensate for the brightness of the face area, correction to make it feel generally preferable, saturation enhancement of the entire image such as color matching and adjustment of luminance distribution, for each input / output device It is not contrary to the gist of the present invention that image correction for compensating for different color reproduction characteristics is performed before or after this processing.

  Hereinafter, each process described above will be described in detail.

(Face region detection processing: S301)
Hereinafter, an example of the face area detection process in step S301 of FIG. 5 will be described in detail.

  Details of the face area detection processing will be described with reference to the flowchart of FIG. FIG. 7 is a diagram for explaining face area detection processing.

  First, in step S401, an area having a low brightness is marked based on the input image. In step S402, two regions marked in step S401 are grouped, and it is determined whether the two groups are eyes from the uniformity of the size of the regions, the difference in luminance, the angle with respect to the horizontal, etc. (FIG. 7 (a)).

  In step S403, it is verified whether or not the face region. That is, for the group determined to be an eye, a rectangular area centering on the two is set (FIG. 7B). Then, the edge near the boundary of the region and the color information in the region are determined. It is determined whether the edge is a face outline. Further, the color information determines whether or not the average value of the RGB values in the rectangular area falls within a preset skin color area ((c) in FIG. 7). In this way, the face area is determined by detecting the position and size of the face. The face area is given by the rectangular area (rectangle parallel to the horizontal / vertical direction).

  In step S404, area information of the determined face area is created. In this embodiment, the area information is a set of position data of the vertices of the circumscribed rectangle of the detected face area, but may be position data of the center of both eyes. Moreover, you may approximate by a polygon, an ellipse, etc.

(Face region discrimination processing: S303)
Hereinafter, an example of the face area determination process in step S303 of FIG. 5 will be described in detail.

  Details of the face area determination processing will be described with reference to the flowchart of FIG.

  First, in step S601, a face area is set in an area of horizontal 48 pixels and vertical 64 pixels so that the center of the left and right eyes is located at the 16th pixel from the left and right ends and the 24th pixel from the top in the horizontal direction. Affine transform. Also, normalization is performed with respect to the luminance distribution and the face orientation. When depth information or 3D information exists in the matching information on the profile, in order to enable feature extraction under the same conditions, the matching should be performed after conversion to the same face direction and light source direction as the correction target. Can improve accuracy.

  In step S602, feature amounts are extracted by the same method as when profile information is created. In step S603, the Euclidean distance from the representative feature amount in each type is obtained on the projected subspace. The partial space is specified by performing a discriminant analysis of the feature amount to obtain a base vector in advance.

  Discriminant analysis is to obtain a projective space that maximizes the ratio of the distance within a class of feature quantities to the distance between classes. Face discrimination using this method is Peter N. Belhumeur, Joao-P. Hespanha, and David J.H. Kriegman et al., “Eigenfaces vs. Fisherfaces: Recognition Using Class Specific Linear Projection” IEEE TPAMI, VOL. 19, NO. 7, JULY 1997 is detailed.

  In step S604, the distance information is normalized from the distance by referring to the profile information conversion table.

  Next, in step S605, it is determined whether or not all the feature values have been collated, and if it is completed, the process proceeds to step S606. If there are still unmatched feature amounts, the process returns to step S603 to repeat the feature amount likelihood calculation. In step S606, a profile ID having the highest likelihood and larger than a predetermined threshold is output and stored.

(Image correction processing: S306)
Hereinafter, an example of the image correction process in step S306 in FIG. 5 will be described in detail. 9 to 13 show the operation of the image correction process. Here, with respect to each face area detected from the above-described image, each skin correction process is performed on the identified area type having a high likelihood.

  Details of the image correction processing will be described with reference to the flowchart of FIG.

  When the image correction process is started, in step S701, one piece of unprocessed face area information detected in step S301 in FIG. 5 is acquired. In step S <b> 702, one unprocessed face type candidate determined in step S <b> 303 in FIG. 5 is acquired from the acquired region with the highest likelihood.

  In step S704, the input image represented by each value in the RGB color space is converted into values of lightness L, saturation C, and hue H. This color space called LCH is calculated from the CIEL * a * b * color space by the following equation (1).

This LCH is an example of a color space that expresses a color in terms of brightness, saturation, and hue. In addition, other color spaces such as YCbCr, L * u * v *, HSV, and HSL may be used.
(Area representative color extraction processing: S705)
In step S705, a representative color in each region is obtained for pixels in the human skin region represented by saturation C and hue H.

  The process of step S705 will be described in detail with reference to the flowchart of FIG.

  First, in step S801, target face image area data is acquired. Steps S802 to S805 constitute a processing loop for each pixel in this region.

In step S802, an unprocessed pixel included in the image area currently focused on is determined. Next, in step S803, it is determined whether the brightness L of the pixel is equal to or greater than a certain value L _TH . If eyes, eyebrows, eyelids, or shadows are included in the face area extracted in step S301, the representative color of the color to be corrected extracted from the human skin area may not be obtained appropriately. Since the shadow in the human skin area or the area such as eyes, eyebrows, and eyelids is considered to have a lightness lower than a predetermined color such as a skin color to be corrected, the shadow in the human skin area can be obtained by using the condition. Alternatively, pixels in the area such as the eyes, eyebrows, and eyelids can be excluded. For the pixel that satisfies the condition in step S803, the color data of the saturation C and hue H of the pixel are totaled in step S804. This aggregation method varies depending on what is used as the representative color to be corrected. For example, when the representative color to be corrected is the average value of the pixels in the human skin region, Mean simple addition of degree C and hue H. At the same time, the number of pixels is counted. The counting process in steps S802 to S805 is repeated until it is determined in step S805 that scanning has been completed for all pixels.

If it is determined that scanning has been completed for all the pixels of the input image, an area representative color of the color to be corrected is calculated for the image area currently focused on in step S806. When the representative color to be corrected is expressed by the average value of the saturation and hue of the pixels in the human skin region, in step S806, a value ( _CR (k, l), H _R (k, l)) is set as the region representative color. Here, k is the index number of the face area, and l is the index number of the type candidate in the discrimination result of each face area. Using the region representative color thus obtained, a correction coefficient for color correction is set in the following step S707.

  Returning to FIG. 9, in step S706, a correction target color set based on the profile ID is acquired.

(Correction coefficient setting process: S707)
In step S707, a correction coefficient for controlling the correction amount for each area is obtained using the area representative color extracted in the previous step S705 and the correction target color acquired in step S706. In the correction coefficient setting process, the correction coefficient is a predetermined coefficient for controlling the color correction amount according to the difference between the region representative color and the target color. Since a correction amount in color correction described later is determined for each hue and saturation, a correction coefficient is also set for each hue and saturation. In this embodiment, the color complementation coefficient and the saturation correction coefficient are set so that the target color decreases as the difference between the representative colors increases. Further, the saturation correction coefficient is set so that the correction amount is smaller when the saturation is corrected so as to decrease the saturation than when the saturation is corrected so as to increase the saturation.

  The correction coefficient setting process in step S707 will be described in more detail below.

The hue correction coefficient P _h (k, l) of the k, l-th (0 ≦ k, l <N) area is the area representative color for the hue HT of the correction target color and the l-th discrimination candidate of the k-th area. The difference H _dist (k, l) between the hues H _R (k, l) is set to decrease as it increases. As an example of a method for setting the hue correction coefficient, a method using an expression such as Expression (2) is shown. However, K ₁ and t ₁ in Equation (2) are constants.

That is, the closer the representative hue of the color to be corrected is to the correction target hue, the closer the hue of the representative color is closer to the hue of the correction target color, and the representative hue of the color to be corrected is far away from the correction target hue. If so, the correction coefficient is set so as not to make much correction. The reason for setting the correction coefficient in this way is that when the representative hue of the color to be corrected is far away from the correction target hue, the correction is performed so that the representative color is closer to the correction target color. Is unbalanced.
Similarly, the saturation correction coefficient l-th determination candidates in k-th region P _C (k, l) is the region of l-th determination candidates in the k-th region and the chroma value C _T of the correction target color The difference C _dist (k, l) between the saturation values C _R (k, l) of the representative colors is set so as to decrease as it increases. In other words, correction is performed so that the saturation of the representative color is closer to the saturation of the correction target color as the representative saturation of the color to be corrected is closer to the correction target saturation, and the representative saturation of the color to be corrected is corrected. When it is far from the target saturation, the correction coefficient is set so as not to make much correction. However, the method of setting the saturation correction coefficient is different depending on whether the correction is performed so as to decrease the saturation or the correction is performed so as to increase the saturation.

When correcting to reduce saturation, it is desirable to set a correction coefficient so as to prevent the phenomenon that the impression of the image becomes plain by correcting, and conversely when correcting to increase saturation This is because, even if the representative saturation of the color to be corrected is far from the correction target saturation, the effect of improving the impression of the image can be expected by performing correction so as to approach the target saturation. As an example of a method for setting such a saturation correction coefficient, a method using a setting equation such as equation (3) in the case of correction for reducing saturation and a method using equation (4) in the case of correction for increasing saturation is shown. It is. It is assumed that a ₁ and b _{1 in} equation (3) and K ₂ and t ₂ in equation (4) are constants.

FIG. 11 shows a graph representing the expressions (3) and (4). In FIG. 11, the horizontal axis represents the difference C _dist (k, l) between the saturation value CT of the correction target color and the saturation value C _R (k, l) of the representative color, and the vertical axis represents the saturation correction coefficient P _C. (k, l) is shown.
Even when the difference between the correction target saturation and the saturation of the representative color is equal, the saturation correction coefficient in the correction for decreasing the saturation is different from the saturation correction coefficient in the correction for increasing the saturation, and the correction for decreasing the saturation. It can be seen from FIG. 11 that the correction coefficient at is smaller. This indicates that when the correction is made to decrease the saturation, the correction amount is made smaller than when the correction is made to increase the saturation.

  As described above, the hue and saturation correction coefficients calculated in step S707 are used as coefficients for controlling the color correction amount used in the correction process (step S710) described later.

(Color area setting processing: S708)
Returning to FIG. 9, in step S708, color area setting processing is performed in order to obtain a color area for each area to be subjected to correction processing including area representative colors. Here, attention is paid to the hue of the pixels in the human skin region.

  In the present embodiment, a color area including a representative color to be corrected is set, and color correction is performed for the area. This is because, for example, if correction is performed to make the skin color a preferable skin color for the entire area of the input image, there is a problem that the colors of other areas in the image that were originally preferable colors become unnatural. It is. The color area to be corrected set in the correction coefficient processing in step S707 is indicated by a specific hue, and color area weights are set according to the correction range indicated by the specific hue and the representative color of the color to be corrected, which will be described later. Used for color correction processing.

  The processing in step S708 will be described in detail using the flowchart in FIG.

  First, in step S1001, face area data belonging to the k-th image area extracted in step S707 of FIG. 9 is acquired.

  Steps S1002 to S1006 constitute a processing loop for each pixel in this region.

In step S1002, an unprocessed pixel included in the currently focused image area is determined. Next, in step S1003, lightness L of the pixel to determine whether there are L _TH than to indicate any brightness. If eyes, eyebrows, eyelids, or shadows are included in the human skin area extracted in step S707 of FIG. 9, the representative color of the correction target extracted from the human skin area cannot be obtained appropriately. There is a case. Since the shadow in the human skin area or the area such as eyes, eyebrows, and eyelids is considered to have a lightness lower than a predetermined color such as a skin color to be corrected, the shadow in the human skin area can be obtained by using the condition. Alternatively, pixels in the area such as the eyes, eyebrows, and eyelids can be excluded.

In step S1004, it is determined whether or not the saturation C of the pixel is equal to or greater than a predetermined value C _TH indicating the saturation of the achromatic color. At this time, the fact that the saturation of the pixel is equal to or less than the above-mentioned value C _TH means that the pixel is close to an achromatic color. Here, the reason for excluding the pixels in the achromatic color region is that in the achromatic color region, the hue changes greatly due to a slight difference in RGB values, and the hue region corresponding to the human skin region cannot be obtained with high accuracy. In the color area setting process in step S708, the color area is set with the hue of a specific color, paying attention to the hue of the pixel in the human skin area.

For the pixel that satisfies the condition of step S1004, the value of the hue H of the pixel is tabulated in step S1005. This aggregation method varies depending on what is used as a specific hue indicating a color area. For example, the hue distribution of pixels that exist in the human skin area and satisfy the conditions of steps S1003 and S1004 is analyzed, and the maximum A method of indicating a color area to be corrected by the hue H _max (k, l) and the minimum hue H _min (k, l) is exemplified. The counting process is repeated until it is determined in step S1006 that scanning has been completed for all pixels.

If it is determined that scanning has been completed for all pixels of the input image, a color region to be corrected is set in step S1007. When the maximum hue H _max (k, l) and the minimum hue H _min (k, l) indicate the color area to be corrected, in step S1007, the maximum hue H _max (k , L) and the minimum hue H _min (k, l), the color area to be corrected is set from the difference between the color to be corrected and the representative hue (formula (5)). Here, H _R (k, l) in the equation (5) is the region representative hue of the color to be corrected of the l th discrimination candidate in the k th region, and ΔH (k, l) is the correction target. The hue range of the color area in the kth image area is shown.

Further, the present invention is not limited to the color area setting process shown here, and for example, a method for manually setting an arbitrary specific hue may be used. In this case, it is desirable to set the color region to be corrected as a hue range that includes the human skin region extracted in step S301 in FIG.
For the color area set by the method as described above, a color area weight to be used in color correction processing described later is determined for each hue range and saturation range. The hue range ΔH indicating the color area to be corrected so that the weight W _H (k, l) in the hue H of the pixel to be corrected is 1 for the representative color to be corrected and 0 outside the correction target area. Based on (k, l) and the hue H _R (k, l) of the representative color of the color to be corrected, it is determined as follows (Formula 6).

However, the method of setting the weight of the hue area to be corrected is not limited to the method represented by the equation (6), and for example, the hue range ΔH (k, l) indicating the color area to be corrected is used. Further, a method of setting the color to be changed nonlinearly using the representative color H _R (k, l) of the color to be corrected and an arbitrary constant is exemplified.
As described above, the reason why the color region to be corrected is weighted in step S1008 is to prevent the color continuity from breaking inside and outside the hue region to be corrected when performing color interpolation described later. It is done.

The weight W _C (k, l) in the chroma C of the pixel to be corrected is by the corrected target color saturation C _T and the predetermined value C _TH showing the saturation of the achromatic used in step S1004, the It is determined as follows (Formula 7).

However, the method of setting the weight of the saturation region to be corrected is not limited to the method represented by the equation (7), and for example, the saturation C _R of the representative color of the color to be corrected and the person A saturation range ΔC (k, l) that includes the skin region is obtained by the same method as the hue region setting method, and the weight of the saturation region to be corrected is set.
Returning to FIG. 9, next, in step S709, the image area weight used for the color correction process described later is set from the position information of the human skin area in the input image.

  In the present embodiment, in a color correction process described later, color correction is performed only for a predetermined color such as a skin color using a weight for the specific color area set in step S708, so that an originally preferable color is obtained. This avoids the phenomenon of unnatural colors in other areas of the image. However, when only the color area is set as the correction target, for example, in the corrected input image, if a pixel other than the human skin included in the color area set in step S708 exists outside the human skin area, the pixel is color-interpolated. It becomes the target of. As a method for solving such a problem, the image area weight set in step S709 is used.

Wherein the image region weight, the skin and the color correction target pixels in the region, the human skin is not included in the region pixel color correction excluded to become as 0 above, the value of 1 or less W _P (k, l, x, y). At this time, it is desirable to set the image area weight so as to prevent color continuity from breaking near the boundary of the human skin area by a color correction process described later.

  The image area weight setting method is not limited, and examples include a method of obtaining a centroid in the human skin area and assigning a weight according to the distance of each pixel from the centroid. When this setting method is used, in the color correction processing described later, the correction amount of pixels near the boundary of the human skin region is smaller than that of the pixel near the center of the human skin region, and the color near the boundary of the human skin region Can be prevented from failing. In addition, there is a method in which the human skin area is subjected to low-pass processing using a Gaussian filter or the like, and the vicinity of the boundary of the human skin area is blurred, thereby relieving the failure of color continuity inside and outside the human skin area.

(Correction process: S710)
Next, in step S710, image correction is performed according to each area correction information obtained previously. This process will be described in detail with reference to the flowchart of FIG.

  First, in step S1101, face area data belonging to the kth image area extracted in step S707 is acquired.

  Steps S1102 to S1105 constitute a processing loop for each pixel in this region.

First, in step S1102, an unprocessed pixel is determined. Next, in step S1103, hue correction processing is performed. First, the hue correction amount Dh (j) for the representative color to be corrected is calculated. The hue correction amount in the representative color of the color to be corrected is the difference between the target hue HT to be corrected and the representative hue H _R (j) of the color to be corrected, and the hue correction coefficient set in the correction coefficient setting process in step S707. It is represented by the product of P _H (j) (Formula (9)). Next, the hue correction amount Dh is set in the hue region weight W _H (j) and saturation region weight W _C (j) set in the color region setting process in step S9 and in the image region setting process in step S709. The hue of the pixel in the input image is corrected by the amount multiplied by the image area weight W _P (j) (formula (8)). In Equation (8), H represents the hue of the pixel to be corrected before the color correction process, and H ′ represents the hue of the pixel to be corrected after the color correction process.

Similarly, in step S1104, the saturation correction factor Kc (j) for the representative color is calculated from the saturation correction factor Pc (j) obtained in the correction factor setting process in step S707, and the saturation correction factor Kc is set in step S1104. The hue area weight W _H (j), the saturation area weight W _C (j) set in the color area setting process of S708, the image area weight W _P (j) set in the image area setting process of step S709, and the above-mentioned The saturation of the pixels in the input image is corrected by the amount corresponding to the saturation C of the correction target pixel before the color correction processing (Formula (11)).
The saturation correction rate Kc is expressed by the ratio of the difference between the corrected target saturation C _T and the correction target representative saturation C _R (j) and the corrected representative saturation C _R (j) as shown in Expression (12). It is represented by the product of the corrected saturation coefficient Pc (j). In Equation (11), C ′ represents the saturation of the correction target pixel after the color correction process.

Next, it is determined whether or not the above process has been completed for all the pixels based on the determination in step S1105. If completed, the correction process is terminated. If not, the process returns to step S1102.
Returning to FIG. 9, next, in step S711, the color space is inversely converted to the RGB space. In step S712, it is determined whether processing has been performed for all detection areas. If not, the process returns to step S701. If correction plan creation has been completed for all areas, the process ends.

<Another Specific Configuration Example of the Image Processing Apparatus of the Present Embodiment>
FIG. 14 is a block diagram illustrating another specific configuration example of the image processing apparatus according to the present embodiment. The same number is given to the part which is common in the said embodiment.

  The block configuration is different from the above embodiment in that a CD / DVD drive 1101 is connected to the drive interface 204 and an optical disc 215 such as a DVD or CD in which the program of this embodiment is recorded is added.

  When the optical disk 215 on which the program of the present embodiment is recorded is inserted into the CD / DVD drive 204, the CPU 201 reads the program from the recording medium and develops it in the RAM 203, whereby the same processing as in the above embodiment can be realized.

<Example of Configuration of Image Processing System of Present Embodiment>
FIG. 15 is a diagram showing a configuration example of an image processing system according to this embodiment in which the image correction method of the present invention is applied to a network system.

  In FIG. 15, a plurality of users (order terminals 1501a, 1501b, 1501c...) And a center server 1502 installed at a remote location with respect to the plurality of users can be communicated through a network 1504. Can transmit desired image data to the center server 1502 via the network 1504, and the image input means of the center server 1502 receives the transmitted image and issues a print instruction to the print shop 1503. It is a figure which shows the form of. The ordering terminal 1501 can be realized by a personal computer owned by an individual. Note that the ordering terminal 1501 may be a portable terminal. In addition, print shops 1503 having ordering, printing, and photographing functions are installed in various places and connected to the center server 1502.

  FIG. 16 is a diagram showing a flow of information communication from when a user orders a print service until an image is output to a print shop. Among these, the center server 1502 performs image correction processing corresponding to the above embodiment.

  First, the homepage of the print service is accessed from the ordering terminal 1501 (S11). The center server 1502 returns a print service input screen (S21). After the login (S12, S22) processing, an image is transmitted from the ordering terminal 1501 (S13), and the center server 1502 receives and stores and holds the image corresponding to the user ID (S23). The profile is transmitted from the ordering terminal 1501 (S14), and the center server 1502 receives and stores the profile corresponding to the user ID (S24). A printing method and a printing instruction are transmitted from the ordering terminal 1501 (S15), and the center server 1502 corrects the image based on the profile received and held corresponding to the user ID (S25). In FIG. 16, image transmission (S13), profile transmission (S14), and print instruction (S15) are shown in a series of procedures, but even if these processes are reversed, there is a long interval between the processes. It may be free, in which case each process starts from login.

  The center server 1502 generates print order data and transmits it to the order terminal 1501 (S26). It is desirable to transmit the corrected image together with the order data for user confirmation. The user confirms the contents of the order, and returns confirmed information from the ordering terminal 1501 to the center server 1502 (S16).

  Upon receiving the order confirmation, the center server 1502 transmits order data to the print shop 1503 (S27), and transmits a print image corrected by the user (S28). Thereafter, the ordering terminal 1501 logs out (S17), and the center server 1502 performs logout processing (S29).

  In the print shop 1503, the order data and the print image are received (S31, S32) and printed out (S33). The printed image is sent to the user who placed the order.

  As described above, the center server 1502 holds profile information corresponding to the user ID, and when a print instruction is received, correction based on the profile information is performed and print processing is performed. Note that the correction information of the profile information held by the center server 1502 may be correction information corresponding to each image or correction information to be applied to a plurality of images according to the user's preference. Correction information selected as corresponding to the user's preference.

  In the present embodiment, the skin color correction process has been described in detail. However, as another correction method, a Gaussian filter may be applied to perform blurring, an edge enhancement filter, or the like. By applying the Gaussian filter to the skin color area around the neck, cheeks and eyes, the apparent age can be made younger. In this case, the degree of effect is defined for each type of face, and image area weight setting similar to that in step S709 is appropriately performed, and an image on which a Gaussian filter is applied according to the weight for each pixel in the area. A weighted average may be obtained for each LHC channel of the original image. Such spatial correction is particularly effective for low resolution images such as moving images and highly portable cameras.

<Other embodiments>
When discriminating the face area, the feature extraction of the face is not limited to the method shown in the first embodiment. For example, a face identification feature descriptor defined in ISO / IEC 15938-3 may be used. Further, as a discriminator, in addition to discriminant analysis, a boost method, a support vector machine combined with a kernel trick, a neural network, or the like may be used. In any case, by preparing a training image for each area type and performing training, it is possible to determine the desired area type. In the present embodiment, the case where the skin color of a person is preferably corrected has been described. However, it is needless to say that the present invention can also be applied to a case where other memory colors such as sky blue and green of plants are preferably corrected.

  Further, as a storage medium for supplying the program code of this embodiment, for example, a floppy disk, a hard disk, an optical disk, a magneto-optical disk, a CD, a DVD, a magnetic tape, a nonvolatile memory card, a ROM, or the like is used. it can.

  Further, by executing the program code read by the computer, not only the functions of the above-described embodiments are realized, but also an OS (operating system) operating on the computer based on the instruction of the program code. It goes without saying that a case where the function of the above-described embodiment is realized by performing part or all of the actual processing and the processing is included.

  Further, after the program code read from the storage medium is written into a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer, the function expansion is performed based on the instruction of the program code. It goes without saying that the CPU or the like provided in the board or the function expansion unit performs part or all of the actual processing, and the functions of the above-described embodiments are realized by the processing.

  Furthermore, in an image input device such as a camera or scanner, an image output device such as a printer, or a device in which these are combined or connected, a CPU or the like provided in both or any of the devices performs part or all of the actual processing. Needless to say, the process includes the case where the functions of the above-described embodiments are realized.

It is a block diagram which shows the structural example of the image processing apparatus of this embodiment. It is a figure which shows the hardware structural example of this embodiment. It is a figure which shows an example of the general view of the profile information storage medium in this embodiment. It is a figure which shows an example of the content of the profile information in this embodiment. It is a flowchart which shows the outline | summary of the operation | movement procedure of this embodiment. It is a flowchart which shows the example of a procedure of the face area | region detection process (S301) of FIG. It is a figure explaining the process example of the face area detection process of FIG. It is a flowchart which shows the example of a procedure of the face area | region determination process (S303) of FIG. 6 is a flowchart illustrating a procedure example of image correction processing (S306) in FIG. 10 is a flowchart illustrating a procedure example of region representative color extraction processing (S705) in FIG. It is a figure explaining the process example of the area | region representative color extraction process of FIG. 10 is a flowchart illustrating an exemplary procedure of color area setting processing (S708) in FIG. 10 is a flowchart illustrating a procedure example of correction processing (S710) in FIG. 9. It is a figure which shows the other hardware structural example of the image processing apparatus of this embodiment. It is a figure which shows the structural example of the network system of this embodiment. It is a figure which shows the flow of the image correction process in the system of FIG.

Claims (9)

Image input means for inputting image information;
Area setting means for setting a face area including a human face in the image information input from the image input means;
Profile information storage means for storing, for each person, profile information including face information representing the characteristics of the person's face and image correction information including target colors corresponding to the face image of the person;
Collating means for collating face information in the profile information for the face area set by the area setting means;
And an image correcting means for performing image correction on the facial region by the smaller amount of color correction is large difference between the representative color of the face area and the target color in the profile information including the matched face information by collation of the collating means An image processing apparatus.   The image processing apparatus according to claim 1, wherein the face information includes one or more face images of each person.   The image processing apparatus according to claim 1, wherein the face information includes a face image having one or more three-dimensional shapes and surface information of each person.   The image processing apparatus according to claim 1, wherein the face information includes irreversible feature data for determining an individual calculated by a predetermined method from the face area of each person.   The image processing according to claim 1, wherein the image correction is any one or a combination of brightness, hue, saturation, sharpness, and blurring for pixels included in the specific color gamut in the specific space region. apparatus. Multiple user terminals;
The image processing apparatus according to any one of claims 1 to 5, wherein the image processing apparatus is installed at a remote location with respect to the plurality of user terminals, functions as a server, and can communicate with the plurality of users via a network. An image processing apparatus that corrects desired image data transmitted by the user via the network according to the profile information transmitted by the user and transmits the corrected image data to the print shop;
A print service system comprising: a print shop that receives and prints the corrected image data in accordance with a print instruction from the image processing apparatus. For each person, and face information representing a feature of the face of the person thereof, stores the profile information and an image correction information including the target color corresponding to the image of the face of the person thereof, the input image based on said profile information An image processing method for performing
An image input process for inputting image information;
A region setting step for setting a face region including a human face in the image information input in the image input step;
A collation step for collating the face information set in the region setting step with face information in the profile information;
And an image correcting step of performing image correction on the facial region by the smaller amount of color correction having a large difference between the representative color of the target color and the face area in the profile information including the face information that matches the collation in the collation step An image processing method comprising:   A computer-executable program for realizing the image processing method according to claim 7.   A storage medium for storing the program according to claim 8 in a computer-readable manner.

Images