JP6958641B2 - Image processing system, image processing method and program - Google Patents

Description

The present invention relates to an image processing system for modifying an image, an image processing method, and a program for causing a computer to execute the method.

The image obtained by capturing the image with a digital camera or the like is modified (corrected) to solve the problem by modifying the defective part. This modification is performed by changing the image expression such as color tone and sharpness, for example.

For image modification, the user or device specifies the means and parameters of image processing, and specifies the type (hereinafter referred to as the standard type) that performs modification according to it, and the target image, feature amount, etc. There are two types of types (hereinafter referred to as transfer types) that are modified toward that goal. In addition to the classification according to the type of such modification, there are the following three types of image modification, which are classified according to the content specified by the user.

The first type is a type in which the user directly specifies an image modification operation (operator) and parameters (hereinafter, referred to as a means designation type). The second type is a type in which the user specifies a target in an image format or the like (hereinafter, referred to as a target specification type). The third type is a type in which the user finally specifies a modified image that matches the expectation from a plurality of modified images (hereinafter, referred to as a result specification type).

Considering the combination of the former two types and the latter three types, a total of six categories can be considered. However, since there is no method of simultaneously satisfying the transfer type and the means-designated type, there are actually five methods.

Examples of the standard type include tone curve conversion, sharpness conversion, edge enhancement, and the like. Since these means are known, detailed description thereof will be omitted. Since the parameters of these means can be freely changed, the degree of freedom in processing is high. However, trial and error is required to achieve the goal, and skill is required. As an example of combining this standard type and the means-designated type, there is a technique in which a user specifies a designated pixel color and a desired color thereof, and the color of the entire image is adjusted based on the specified pixel color (see Patent Document 1). ).

Further, there is also a technique of creating a correspondence relationship between pixel values before and after adjustment as a color conversion list using the adjustment result of image data, and applying this to another image for correction (see Patent Document 2). In addition, there is a technique for easily creating a color conversion list including user interaction (see Patent Document 3) and a similar technique for assuming an image as a processing target (see Patent Document 4).

As an example of combining the standard type and the target specification type, there is a technique executed by an application called Instagram, in which the effect is presented to the user in an icon format and selected. As an example of combining the standard type and the result specification type, there is a technique in which a small number of typical processes and parameters are determined in advance, and the modified image is presented to the user as a pair and selected (Patent Document 5). reference).

As an example of combining the transfer type and the target specification type, the image characteristics (highlight density, shadow density, average density, etc.) are used as the reference image in order to align the color gradation of the finished photographic print of similar scenes. There is a technique for matching the characteristics (see Patent Document 6). In addition, in order to match the color gradation of the face of the same subject with a digital photo printer, the user is allowed to select the target face image (area), and the color gradation of the color region of the face image of the input image is set to the target face. There is also a technique for matching the color gradation of the color region of an image (see Patent Document 7). Furthermore, in order to intuitively reproduce the desired image, there is also a technique for correcting the input image so that it approaches the image reproduction item of the teacher data referred to by the image reproduction item (color reproduction, gradation reproduction, contour reproduction) (patented). Reference 8).

As an example of combining the transfer type and the result designation type, from the result of correction processing so that the second feature amount of the second image to be image-corrected is brought closer to the first feature amount of each of the plurality of types of first images. , There is a technique for selecting a desired first image (see Patent Document 9).

The combination of the standard type and the means-specified type sacrifices the processing degree of freedom, which is an advantage of the standard type, and gives the user a simple choice, so that the processing degree of freedom is limited. In addition, it is difficult to predict what kind of change the result of applying the color conversion for each pixel will bring about the impression of the entire image until it is actually executed. In this respect, the combination of the standard type and the target specification type is also difficult to predict because the actual effect depends on the input image, and trial and error is required. The combination of the standard type and the result specification type can be selected from the modified images, so there is no problem in predictability, but the degree of freedom in processing is limited because the user is given a choice at the expense of the degree of freedom in processing. It ends up.

Since the means and parameters cannot be freely changed in the combination of the transfer type and the target specification type, the degree of freedom in processing is extremely limited. Difficulty in predicting is also alleviated by each of the above combinations, but it is not resolved. The combination of the transfer type and the result specification type can eliminate the difficulty of prediction, but like the combination of the transfer type and the target specification type, the problem that the degree of freedom of processing is limited cannot be solved. Can not.

In this way, the combination of the two types can eliminate the difficulty of prediction as compared with the standard type alone, but the degree of freedom in processing is limited. If only the standard type is used in order to increase the degree of freedom in processing, it will be difficult to predict. By the way, if the prediction difficulty does not occur, the intuition is enhanced, the usability is good, and the desired result can be reached with less man-hours. Therefore, it has been desired to provide a system and a method that are intuitive and easy to use, can reach a desired result with less man-hours, and can increase the degree of freedom in processing.

In view of the above problems, the present invention is an image processing system for modifying an image, and identifies the reference information at the time of modification referred to for specifying the content of the modification and the case where the modification is performed according to the content of the modification. An information storage unit that stores a plurality of case information including possible case identification information, an image acquisition unit that accepts an image input and acquires the image, and a plurality of case information stored in the information storage unit, respectively. It is specified by the case selection unit that arranges and displays on the display unit based on the case identification information included in the case information and accepts the selection of the case information from the user, and the reference information at the time of modification included in the selected case information. An image processing system is provided that includes a case application unit that modifies the image acquired by the image acquisition unit according to the content of the modification.

According to the present invention, it is intuitive and easy to use, a desired result can be achieved with less man-hours, and the degree of freedom in processing can be increased.

The figure which showed the configuration example of the image processing system of this embodiment. The hardware configuration diagram of the image processing system of this embodiment. The figure explaining the type of image modification. The figure explaining the case information. The figure which showed the example which hierarchized the case information of the standard type and the transfer type. The figure which illustrated the screen for selecting one case information from a plurality of case information. The figure which showed an example of the screen when the case information was arranged and displayed based on the case identification information. The figure which showed another example of the screen when the case information was arranged and displayed based on the case identification information. The figure which showed the display example of the goal specification type and the result specification type. The functional block diagram which showed the 1st Embodiment of the image processing system of this embodiment. The flowchart which showed the flow of the image processing executed by the image processing system shown in FIG. The functional block diagram which showed the 2nd Embodiment of the image processing system of this embodiment. The figure which illustrated the screen which selected the area to be retouched by the area selection part. The flowchart which showed the flow of the image processing executed by the image processing system shown in FIG. The functional block diagram which showed the 3rd Embodiment of the image processing system of this embodiment. The flowchart which showed the flow of the image processing executed by the image processing system shown in FIG. The functional block diagram which showed the 4th Embodiment of the image processing system of this embodiment. The figure which showed the example of the screen which displayed the evaluation index which makes a user evaluate by an image evaluation part. The flowchart which showed the flow of the image processing executed by the image processing system shown in FIG.

FIG. 1 is a diagram showing a configuration example of an image processing system. The image processing system includes an image processing device 10 and a server device 11 that stores case information described later. The image processing system may be composed of only the image processing device 10 as long as the case information is not stored in the server device 11 but stored in the storage device included in the image processing device 10. Further, the case information can be stored in a device other than the image processing device 10 and the server device 11, and in this case, the image processing system includes the image processing device 10 and the device thereof.

In the example shown in FIG. 1, in addition to the image processing device 10 and the server device 11, an image reading device 12 that reads a document or the like and inputs an image as image data to the image processing device 10 and an image processing device 10 that captures an image of a subject. It includes an image pickup device 13 for inputting an image as image data to. In addition to the above case information, the server device 11 can store image data as an image, and can read and transmit the image data at the request of the image processing device 10.

In the image processing system shown in FIG. 1, the image processing device 10 and the image pickup device 13 are directly connected by a cable 14, and the image processing device 10, the image reading device 12, and the server device 11 are connected to each other via a network 15. There is.

The image processing system may be configured to include two or more image processing devices 10, a server device 11, an image reading device 12, and an image pickup device 13, or may be configured to further include other devices. In FIG. 1, the cable 14 is used for connection, but it can also be connected by wireless communication. Further, the image processing device 10, the image reading device 12, and the server device 11 may be directly connected by a cable instead of the network 15.

Further, the network 15 can be connected by a cable or wirelessly. When connecting wirelessly, it is possible to connect to the network 15 via a base station called an access point. The network 15 can use a WAN (Wide Area Network), the Internet, or the like.

As the image processing device 10, a PC, a tablet terminal, or the like capable of modifying an image can be used. The hardware configuration, functions, and processing contents of the image processing device 10 will be described later.

As the server device 11, a PC, a workstation, or the like can be used. The hardware configuration of the server device 11 can be the same as the hardware configuration of the image processing device 10. Therefore, the description thereof is omitted here. The image processing device 10 can access the server device 11, request case information and image data from the server device 11, and acquire the requested case information and image data from the server device 11.

As the image reading device 12, a scanner device, an MFP (Multi Function Peripheral), or the like can be used. The image reading device 12 includes a platen for setting a document at a predetermined position, a light source for irradiating the document with light, a mirror for guiding the light from the document in a predetermined direction, and a photoelectric conversion element for converting light into an electric signal. , A / D converters and the like that convert electrical signals into digital data. The image reading device 12 outputs the converted digital data as image data to the image processing device 10.

As the image pickup device 13, a digital camera, a video camera, or the like can be used. For example, a digital camera has an optical system including a lens and an aperture mechanism, a light receiving element that receives light incident through the lens and outputs it as an electric signal, and processes such as converting the output electric signal into digital data. Includes an image calculation circuit that performs Further, the digital camera includes a memory for recording the processed digital data as image data, a display device for displaying the captured image, a strobe, an operation switch, an interface for connecting a cable, and the like. The image processing device 10 can access this memory and read and acquire the image data recorded in this memory.

The hardware configuration of the image processing apparatus 10 will be described with reference to FIG. Similar to a general PC, the image processing device 10 includes an input I / F 20, a CPU 21, a storage device 22 such as a memory, an auxiliary storage device 23 such as an HDD, an output I / F 24, a drive 26 into which a recording medium 25 is inserted, and a controller. 27 is provided. The input I / F 20 connects, for example, one end of the cable 14 when the image pickup device 13 is connected to the image processing device 10 by the cable 14. The other end of the cable 14 is connected to the image pickup device 13.

The CPU 21 executes a program stored in the auxiliary storage device 23, controls the image processing device 10, and realizes a predetermined function. By realizing this predetermined function, the image is modified. The storage device 22 provides a working space for the CPU 21 and stores a boot program, firmware, setting parameters, and the like. When the output I / F 24 is connected to a display device (not shown) by a cable, one end of the cable is connected. The other end of the cable is connected to a display device (not shown).

On the recording medium 25, for example, a program for modifying the above image is recorded. This program is installed in the auxiliary storage device 23 via the drive 26 that controls reading and writing of the recording medium 25. The controller 27 functions as a network I / F connected to the network 15 and controls communication with the network 15. The above program may be independent program software or may be implemented in the printer driver.

Before explaining the specific function of the image processing apparatus 10 and the processing executed by the function, the outline of the present invention will be described. The drawback of the standard type, which is modified using the specified image processing means and parameters, is that it requires skill and trial and error. The disadvantage of the combination of the standard type and the means-designated type, or the combination of the transfer type and the target-designated type that modifies the image to be referred to as the target is that the degree of freedom of processing is low. These points are correlated, and it has not been possible to eliminate both drawbacks with conventional techniques.

In order to eliminate both of these drawbacks, the present invention introduces a mechanism in which the particle size (unit) of image modification is made larger than that of the above standard type, and they can be registered and reused. Then, the unit that can be registered and reused is defined as an example. The example is an example of the modification performed by a skilled person, and from the example, case information such as the content of the modification, the parameters used, and the images before and after the modification can be obtained. By applying such a case, it is possible to modify the desired image without requiring skill and trial and error. Further, if a desired case can be selected and applied from many cases, the degree of freedom in processing can be increased.

Therefore, it is necessary to collect cases, but the transfer type is effective when considering the efficient collection of cases. This is because it is only necessary to collect the desired image, and it is not necessary to record the input image or the modification process in the middle. However, considering the following two points, it is not a good idea to exclude the standard type. In other words, many professional retouchers (professional photo editors) actually utilize standard-type retouching, and transfer-type retouching is still under development.

More specifically, standard image modification can only be properly used by a skilled person. Experts have a lot of knowledge and experience and understand what kind of image processing means should be applied, in what order, and what kind of result should be obtained. Experts can respond according to the pattern of the image, have an aesthetic eye to evaluate the quality of the modified image (modified image), and have these skills comprehensively, so the standard type image Allows retouching. However, such skills are extremely valuable and difficult to easily externalize. For this reason, the current situation is that beginners and inexperienced people acquire skills by repeating trial and error while being taught by experts. Moreover, it is difficult for even an expert to express and convey clearly in words.

Therefore, the only way to externalize such skills is to accumulate and utilize examples of image modification performed by experts. In the present invention, by introducing an information format called a case, a method selected by an expert is accumulated and used. Then, the methods and images accumulated in the standard type can be used in the transfer type.

Here, with reference to FIGS. 3 to 5, modifications and case information for each type will be described. FIG. 3A is a diagram illustrating modification by the standard type. In this modification, the two images before and after the modification used in the case (case image) and the tone curve which is the means of image processing applied in the case are used. The tone curve is a means for correcting the gradation of an image, and the image A of the two case images is gradation-corrected so as to become the other image B. Therefore, when the image to be modified is input as an input image, it is converted by this tone curve, and the converted image is output as a modified image. The tone curve can be represented by a gradation conversion function, which can be used for conversion. In FIG. 3A, the background, the color of the person's face, the color of clothes, and the like are changed by this modification.

FIG. 3B is a diagram illustrating modification by the transfer type. In this modification, only the image referred to as the target (reference image) is used. This is because in the transfer type, only the reference image is required because the modification is performed toward the reference image. This transfer type can be modified using known methods such as color transfer (http://www.thegooch.org/Publications/PDFs/ColorTransfer.pdf). In FIG. 3B, the background, the color of the person's face, and the color of the clothes are changed by this modification.

FIG. 4A is a diagram showing an example of standard type case information, and FIG. 4B is a diagram showing an example of transfer type case information. The case information includes the reference information at the time of modification that is referred to for specifying the content of the modification, and the case identification information that can identify the case in which the modification is performed according to the content of the modification. In FIG. 4, an image processing operator and an image processing parameter are shown as reference information at the time of modification.

The image processing operator represents a means of image processing, and the operator name “L tone curve” is shown in FIG. 4A, and the operator name “transfer operator” is shown in FIG. 4B. The image processing parameter is a parameter used for the means, and in FIG. 4A, the control point information used to create the tone curve is shown, and in FIG. 4B, there is no parameter, so NULL is shown. ing. In addition, in FIG. 4B, following NULL, it is described in parentheses as "Refer to image B", and it is shown that image B is used as a reference image.

The case identification information may be any information as long as it can distinguish the case from other cases. In FIG. 4, the type that is the type of modification, the case identifier (case ID), the shooting position, the shooting date and time, the image file name, the color tone information, the scene name of the shooting scene, the name of the object to be shot, and the impression representing the impression of the image are shown. The selection area information when selecting the area in the expression and the image is shown. Therefore, the above-mentioned reference information at the time of modification can also be included in the case identification information if it can be distinguished from other cases.

Since FIG. 4A is a standard type, the type is a standard type, the latitude and longitude information is used as the shooting position of the image, the name of the person in the image is used as the object name, and the impression information of the person is used. The impression is shown. Further, in the case of the standard type, since two images are required as case images, the file names and color tone information of the images A and B are registered.

Since FIG. 4B is a transfer type, the type is a transfer type, and other than that, it is the same as that of FIG. 4A. However, in the case of the transfer type, since only the reference image needs to be provided, the file name and color tone information of only the image B, which is the reference image, are registered. Therefore, in the other file name and color tone information fields, "Null" indicating that there is no information is input.

Here, the case information is expressed in a table format, but in reality, it can be realized according to a description method such as XML (eXtensible Markup Language). Since this is an example, other description methods may be adopted.

The two case images collected in the standard case can be used as reference images for the two transfer types. The standard type case information shown in FIG. 5 (a) is the same as that in FIG. 4 (a), but the information of image A and the information of image B in this are used as reference information, and FIGS. 5 (b) and 5 (c) are used. ) Can be created as shown in case information. In FIG. 5, case information can be stored in a hierarchical manner, with FIG. 5A as a root node and child nodes connected to the root node as FIGS. 5B and 5C. By doing so, it is possible to provide a framework capable of modifying both the standard type and the transfer type. The arrows in FIGS. 5 (b) and 5 (c) mean that the contents of the corresponding items in FIG. 5 (a) are the same.

Next, in order to realize intuitive and easy usability, it is preferable to visually display the case in some form and give the user a choice. When the number of cases is small, the case information can be displayed side by side as it is.

However, as the collection and registration of cases progresses, even if they are displayed side by side as described above, there is a limit to the number that can be displayed at one time, so it becomes impossible to display all the cases at once. As shown in FIG. 6, since there are several to several tens of case images and reference images as examples that can be displayed at one time, the screen must be scrolled and displayed by a scroll bar or the like. It is very difficult to find suitable case information when the display is not restricted and arranged randomly.

Therefore, instead of including the case identification information in the case information and arranging them randomly, based on the case identification information (for example, arranging them according to the type and color tone information), the case information is appropriately displayed on the display unit which is a display device. Make it possible to arrange and display it. This makes it possible to solve the problem that it is difficult to find suitable case information.

The skin color of the face image will be specifically described with reference to FIG. 7. The case identification information includes selection area information for selecting a specific area such as a face area, and color tone information of the area selected by the selection area information. The color tone information is, for example, an average Lab value. L is a dimension indicating lightness, and a and b are color dimensions, and are represented by coordinate values in the Lab color space as illustrated in FIGS. 4 and 5 (80, 10, 10). This color tone information may be calculated and registered at the time of registration, or may be calculated when needed.

In the transfer type, the modification is performed using the color tone information of the face image in the image B which is the case image. Therefore, by providing a rule to use this color tone information for the standard type or letting the user select it, both can be arranged two-dimensionally regardless of the type of the case. The display may be in an image format such as an icon or a thumbnail, or may be in a language format such as a case ID or a shooting position. Incidentally, in FIG. 7, a list of case information is displayed in an image format.

In addition to the list display of case information as shown in FIG. 7, the display can map the ab plane (two-dimensional) of the Lab space as shown in FIG. 8 and display each point as each case. When the cursor is moved to an arbitrary point and stopped at that point, the case information may be displayed by an icon or the like. Further, the current input image may be shown in a shape that can be distinguished from a point, or in a star shape in FIG. 8, so that how to select a target can be performed more intuitively. In this example, the skin color of the face image is taken as an example, but other scene names, object names, and the like may be used.

When displaying a list of all case information, it is necessary to display as many cases as there are cases, but in many cases, case information is selected within the same category, so by limiting the target of list display to that category, it is displayed. The number of case information to be used can be reduced. For example, the number of case information to be displayed can be reduced by allowing items such as scene names and object names in the case identification information to be selected as categories. In FIG. 8, "sky", "turf", "flower" and the like can be selected by tabs.

In addition, when the type is selected, as shown in FIG. 9A, the standard type must display two case images for each case information, but the transfer type requires only one reference image, so that it can be displayed at once. It is possible to display a lot of case information. In addition, it is possible to specify a goal. As for the standard type, as shown in FIG. 9B, the two case images can be displayed alternately at regular time intervals instead of being displayed side by side. This makes it possible to display more case information at one time, as in the case of the transfer type.

The image to be displayed may be only two case images for the standard type and one reference image for the transfer type, but as shown in FIG. 9 (c), after the modification is performed according to the content of the modification in the case. The modified image of is also displayed. This makes it possible to specify the result. Even in this case, for the standard type, three images including the modified image can be displayed alternately. Also, for the transfer type, two images including the modified image can be displayed alternately. This makes it possible to display more case information at one time.

Although the outline has been described so far, the functional configuration for realizing the above modification and the processing performed by each functional unit will be described in detail below. FIG. 10 is a functional block diagram showing a first embodiment of the image processing device 10. The image processing device 10 includes a control unit 30, an image acquisition unit 31, a case acquisition unit 32, an information storage unit 33, a case selection unit 34, a case application unit 35, and an image output unit 36 as functional units. Consists of including. The case application unit 35 includes a first type correction unit 37 and a second type correction unit 38.

The control unit 30 controls the entire image processing device 10. The image acquisition unit 31 receives input of an image from the image reading device 12, the image pickup device 13, the server device 11, and the like shown in FIG. 1, and acquires the image. Here, it is assumed that an image is acquired, but in reality, the image data of the image is acquired. The image may be acquired by acquiring the image data input from the image pickup device 13 or the like, or the control unit 30 may receive an instruction from the user and request the server device 11 to acquire the read image data. can.

In response to the image acquisition unit 31 acquiring the image, the case acquisition unit 32 acquires a plurality of case information stored in the information storage unit 33 and passes it to the case selection unit 34. The case acquisition unit 32 can also acquire all the case information stored in the information storage unit 33, and when the user selects the above category, it can acquire only the case information of that category. For example, when the user selects the transfer type as the type, only those including the transfer type in the case identification information can be acquired.

The case selection unit 34 arranges and displays a plurality of case information received from the case acquisition unit 32 on the display unit based on the case identification information included in each case information. Then, the case selection unit 34 accepts the selection of the case information from the user. The case application unit 35 specifies the type of modification in the case, that is, the type from the case identification information included in the case information selected by the user. If the type is a standard type, the first type correction unit 37 is requested to make corrections, and if the type is a transfer type, the second type correction unit 38 is requested to make corrections.

Upon receiving the request, the first format correction unit 37 uses a standard type for the image acquired by the image acquisition unit 31 according to the content of the modification specified by the reference information at the time of modification included in the case information selected by the user. Make corrections. Upon receiving the request, the second format correction unit 38 uses a transfer type for the image acquired by the image acquisition unit 31 according to the content of the modification specified by the reference information at the time of modification included in the case information selected by the user. Make corrections.

When the modification is completed and the modified image is obtained, the case application unit 35 passes it to the image output unit 36. The image output unit 36 displays the modified image on the display unit, prints and outputs the modified image, and further stores the modified image in the information storage unit 33. These outputs can be executed according to preset contents.

These functional units are realized by the CPU 21 shown in FIG. 2 reading and executing a program for image modification stored in the auxiliary storage device 23 in the storage device 22. The information storage unit 33 can be a storage device included in the server device 11, but other devices such as an auxiliary storage device 23 included in the image processing device 10 and an external storage device connected to a cable or a network 15. It is also possible to.

The process of modifying the image executed by the image processing apparatus 10 shown in FIG. 10 will be described in detail with reference to the flowchart shown in FIG. In step 1100, the image acquisition unit 31 receives an image input from the image pickup device 13 or the like to start the process. In step 1105, the image acquisition unit 31 acquires the input image. In step 1110, the case acquisition unit 32 acquires a plurality of case information from the information storage unit 33 in response to the image acquisition unit 31 acquiring the image. Here, it will be described as assuming that all the case information stored in the information storage unit 33 is acquired.

In step 1115, the case selection unit 34 arranges and displays the case information acquired by the case acquisition unit 32 on the display unit based on the case identification information included in each case information. The simplest arrangement example is an arrangement in which case information is arranged in the order of case IDs. Since the case ID is composed of numerical values, alphabets, and the like, it can be arranged in order from the number 1 or in alphabetical order. In addition to this, it is also possible to arrange them in the order of shooting position and shooting date and time.

The case selection unit 34 can specify items by user interaction, that is, dialogue with the user, and arrange them in the order of the items, or can request arrangement according to predetermined items, for example, color tone information, and arrange them there. can.

It is necessary to appropriately display the case information in consideration of the difference in the modification type. In the case of the standard type, the two images (case images) before and after the modification are displayed side by side or alternately. In the case of the transfer type, only the reference image referred to as the target is displayed. This makes it possible to specify a goal.

Further, regarding the display method of the case information, it is possible to display the modified image for the image (input image) acquired by the image acquisition unit 31 as well. In the case of the standard type, the modified image is included in the two case images. Display the three images side by side or alternately. In the case of the transfer type, two images including the modified image in the reference image are displayed side by side or alternately. This makes it possible to specify the result.

In step 1115, the user selects suitable case information from the case information displayed in this way, and the case selection unit 34 receives the case information selected by the user. In step 1120, in response to the selection of the case information, the case application unit 35 first refers to the type in the case identification information and identifies the type in order to apply the case information to the input image. I do. If the type is a standard type, the process proceeds to step 1125, and if the type is a transfer type, the process proceeds to step 1130.

In step 1125, the first type correction unit 37 performs the correction by the standard type, and in step 1130, the second type correction unit 38 performs the correction by the transfer type. The modification is performed by taking out the image processing operator and the image processing parameter in the modification reference information included in the case information and applying them. Tone curves, sharpness, edge enhancement, etc. are specified for the image processing operator, and control point information and the like used for them are specified for the image processing parameters. The first form correction unit 37 performs tone curve conversion if a tone curve is specified, and performs sharpness conversion if sharpness is specified. Since these conversions are already known, the description thereof will be omitted. The number of image processing operators is not limited to one, and may be two or more.

The above tone curve and the like are specified for the standard type, and a transfer operator is specified for the transfer type. At this time, a reference image is specified in the image processing parameter. The transfer type modification can be performed by using a known method such as the above-mentioned color transfer. When the image modification is completed, the first format correction unit 37 or the second format correction unit 38 passes the modified image as the modification result to the case application unit 35.

In step 1135, the image output unit 36 receives the modified image from the case application unit 35, displays the modified image on the display unit, and presents the modified image to the user. In step 1140, it is determined whether the presented modified image is acceptable. For example, if the OK button and the redo button are displayed, and the user presses the OK button, it can be determined that the modified image is sufficient, and if the redo button is pressed, the modified image can be determined to be useless.

If the result determined in step 1140 is a modified image, the process proceeds to step 1145, the image output unit 36 stores the modified image in the information storage unit 33, and the process ends in step 1150. On the other hand, if the modified image is not good, the process returns to step 1115, the user selects the case information again, and the case selection unit 34 accepts the selection.

In the embodiment described above, it is possible to modify by two types, but it is also possible to limit the modification to one type and modify only that type. In this case, the case identification information does not have to include the item of type, and the case application unit 35 does not need to make a determination to specify the type, as long as it holds only one of the correction units. good.

Next, another functional configuration of the image processing apparatus 10 and processing performed by each functional unit will be described. FIG. 12 is a functional block diagram showing a second embodiment of the image processing device 10. Similar to the image processing device 10 shown in FIG. 10, the control unit 30, the image acquisition unit 31, the case acquisition unit 32, the information storage unit 33, the case selection unit 34, the case application unit 35, and the image output unit It is configured to include 36 and. The case application unit 35 includes a first type correction unit 37 and a second type correction unit 38. The embodiment shown in FIG. 12 further includes a region selection unit 40. Since the same functional units such as the control unit 30 have already been described, only the area selection unit 40 will be described here.

The area selection unit 40 is also realized by the CPU 21 executing a program, and selects an area to be modified. The area selection unit 40 limits the area to be modified in the input image.

It is possible to modify the entire image based on case information in both the standard type and the transfer type. The standard type is suitable for simple modifications such as brightening and vivid color tones, and the transfer type is suitable for scenes in which the entire image is similar. In the modification for the entire image, there are cases where such a suitable case is used, but on the other hand, there are cases where the modification cannot always be performed on the intended image. For example, when it is desired to brighten only the face a little, when it is desired to suppress only the bluish component, or when it is desired to emphasize only the high frequency component.

When it is desired to modify only such a specific region, the region selection unit 40 can select the region to be modified, and the case application unit 35 can modify the selected region. The region can be selected by using the already known method of selecting by hue, the method of tracing the region boundary by the user, the method of tracing the region surface with a brush, the face region extraction, the fast Fourier transform (FFT), and the like. ..

More specifically, with reference to FIG. 13, the input image 41 is displayed in FIG. 13, and the toolbox 42 is displayed on the left side of the input image 41. Various tools are displayed as icons in the toolbox 42, and here, the automatic selection tool 43 is selected. The automatic selection tool 43 is a tool that automatically selects the pixel at the position clicked by the mouse and the pixel of the approximate color adjacent thereto.

In FIG. 13, since the pixel displaying the skin of the face is clicked, all the pixels of the skin color adjacent to the pixel are extracted, the area formed by all the extracted pixels is selected, and the area formed by all the extracted pixels is displayed by the broken line. ing.

The area selected by the area selection unit 40 is not limited to the area in the image space (plane). A specific channel of the color image, for example the channel region of the image signal represented by red, blue, green, may be selected, or the frequency domain of the image signal having a specific band in the frequency space may be selected. The area selection unit 40 is not limited to selecting one of these three areas, and may select a area in which two or more of these three areas are combined.

The process of modifying the image executed by the image processing apparatus 10 of this embodiment will be described with reference to the flowchart shown in FIG. This process is started from step 1400. In this case as well, as in FIG. 11, the image acquisition unit 31 starts inputting an image from the image pickup device 13 or the like. Since steps 1405 and steps 1415 to 1455 are the same as steps 1105 to 1150 shown in FIG. 11, the description thereof will be omitted here.

In step 1410, the area selection unit 40 receives the designation of the area from the user and selects the designated area. The information of the selected area is sent to the case application unit 35, the area to be modified is specified, and the area to be modified is modified.

Further another functional configuration of the image processing apparatus 10 and processing performed by each functional unit will be described. FIG. 15 is a functional block diagram showing a third embodiment of the image processing device 10. Similar to the image processing device 10 shown in FIG. 10, the control unit 30, the image acquisition unit 31, the case acquisition unit 32, the information storage unit 33, the case selection unit 34, the case application unit 35, and the image output unit It is configured to include 36 and. The case application unit 35 includes a first type correction unit 37 and a second type correction unit 38. The embodiment shown in FIG. 15 further includes a case registration unit 50. Since the same functional units such as the control unit 30 have already been described, only the case registration unit 50 will be described here. The image processing device 10 may include both the case registration unit 50 and the area selection unit 40 described above.

The case registration unit 50 is realized by the CPU 21 executing a program, adds additional information to the modified image obtained by image modification, stores it in the information storage unit 33 as new case information, and registers it. The case registration unit 50 identifies the content corresponding to each item according to the expression format shown in FIG. 4, and registers it as one case information. Therefore, the content corresponding to each item, excluding the modified image, is used as additional information. As for the type, the type specified at the time of modification is inherited as it is.

When the type is the standard type, the image processing operator and the image processing parameter referred to at the time of modification are specified as they are in the reference information at the time of modification. As the case identification information, the contents of the input image and the modified image are designated as the contents of the image A and the image B shown in FIG. That is, the file names and color tone information of the input image and the modified image are specified. For other information, each information in the case information referred to at the time of modification is specified as the default value. Therefore, if correction is necessary, the information of the portion to be corrected can be corrected via user interaction.

Even when the type is transfer type, the image processing operator and the image processing parameter referred to at the time of modification are specified as they are in the reference information at the time of modification. Then, in the case identification information, the content of the modified image is designated as the content of the image B. Other information is the same as the standard type.

By creating one new case information in this way and adding it, the case information can be reused. In the case of the standard type, as shown in FIG. 5, since two case images are included, each of the two case images can be used as a reference image and registered as two transfer type case information. As a result, in addition to the standard type case information, the transfer type case information can be added. Further, by adding the case information, it becomes possible to obtain a more desired modified image.

The process of modifying the image executed by the image processing apparatus 10 of this embodiment will be described with reference to the flowchart shown in FIG. This process is started from step 1600. In this case as well, as in FIG. 11, the image acquisition unit 31 starts inputting an image from the image pickup device 13 or the like. Since steps 1605-step 1645 and 1655 are the same as steps 1105-step 1150 shown in FIG. 11, the description thereof will be omitted here.

In step 1650, the case registration unit 50 sets the modification made to the input image as an additional case, and stores and registers the case information obtained in that case as additional case information in the information storage unit 33. This registered case information is presented to the user and made selectable in the next image modification.

Further another functional configuration of the image processing apparatus 10 and processing performed by each functional unit will be described. FIG. 17 is a functional block diagram showing a fourth embodiment of the image processing device 10. Similar to the image processing device 10 shown in FIG. 10, the control unit 30, the image acquisition unit 31, the case acquisition unit 32, the information storage unit 33, the case selection unit 34, the case application unit 35, and the image output unit It is configured to include 36 and. The case application unit 35 includes a first type correction unit 37 and a second type correction unit 38. The embodiment shown in FIG. 17 further includes an image evaluation unit 60. Since the same functional units such as the control unit 30 have already been described, only the image evaluation unit 60 will be described here. The image processing device 10 may include an image evaluation unit 60, and / or both of the area selection unit 40 and the case registration unit 50.

The image evaluation unit 60 is realized by the CPU 21 executing a program, and displays an index for evaluating the quality of the modified image on the display unit. If you are not an expert, it is difficult to judge whether or not the desired image was obtained just by looking at the modified image because the discerning eye is not sufficient. Therefore, it is desirable to provide an image evaluation unit 60 to display an evaluation index that allows the user to objectively evaluate.

The image evaluation unit 60 utilizes at least one of image pair evaluation and single evaluation in order to provide an evaluation index. The evaluation index displayed by the image evaluation unit 60 will be described with reference to FIG. The image shown in FIG. 18 is an input image, a modified image, and a Ch difference (brightness L) image from the left side, and the Ch difference image is an example of an evaluation index obtained by image pair evaluation. ΔE (color difference) is shown as a numerical value on the lower side of the Ch difference image. This ΔE is also an example of an evaluation index obtained by image pair evaluation. In short, the difference between single evaluation and image vs. evaluation is whether the evaluation index is obtained from a single image or from two images.

S / N (signal-to-noise ratio), MSE (mean squared error), ΔE, and the like are known as examples of evaluation indexes obtained by image pair evaluation. With these evaluation indexes, it is possible to objectively understand how much the signal level changes. In addition, an evaluation index called SSIM, which is said to have a higher degree of consistency with subjectivity than S / N or MSE, can be introduced.

As an example of the evaluation index obtained by a single evaluation, a histogram for each lightness or color channel is known. This histogram is, for example, a bar graph showing how many pixel values of R, G, and B constituting the image are used. Therefore, by looking at this histogram, the user can objectively understand the tone of the entire image and make an appropriate judgment. Factors for evaluating image quality include contrast, sharpness, sharpness, graininess, etc., in addition to brightness and the like, and evaluation indexes for these can be introduced.

Other examples of indicators obtained by a single evaluation include Saliency Map (attractiveness map). This is a method of expressing the degree of conspicuity in the light of human visual characteristics as an image by diverting pixel values. With this map, you can get guidance on which hits to focus on and evaluate. These examples are just examples, and other evaluation indexes may be added.

Examples of the image or image pair to be evaluated include a modified image, a modified image and an input image, a reference image, or a case image. The image or image pair to be evaluated may be another input image, a reference image, an input image and a reference image, or the like. Of these, the image pair evaluation of the input image and the reference image or the case image will be the basis for expanding to the purpose of arranging more suitable case information in the future. The evaluation of the modified image is used as objective information for judging the obtained result. With this evaluation, the user can make an evaluation with peace of mind, and at the same time, it is possible to improve the skill level required for image modification.

The process of modifying the image executed by the image processing apparatus 10 of this embodiment will be described with reference to the flowchart shown in FIG. This process is started from step 1900. In this case as well, as in FIG. 11, the image acquisition unit 31 starts inputting an image from the image pickup device 13 or the like. Steps 1905 to 1935 and steps 1945 to 1655 are the same as steps 1105 to 1150 shown in FIG. 11, and thus the description thereof will be omitted here.

In step 1940, the image evaluation unit 60 causes the user to display an evaluation index in order to evaluate the quality of the modified image. The image evaluation unit 60 generates an index called the above-mentioned histogram using the information of the modified image, for example, the pixel value of RGB, and displays the quality of the modified image on the display unit in order to present the histogram to the user. Further, the image evaluation unit 60 obtains the above S / N or the like using the modified image and the information of the input image, the reference image, or the case image, and presents the numerical value as an index to the user. Display it.

As described above, by arranging and displaying the case information based on the case identification information, it is intuitive and easy to use, the desired result can be reached with less man-hours, and from many cases. By making it selectable, the degree of freedom in processing can be increased. In addition, since the case identification information includes the color tone information, the scene name, and the like and is appropriately arranged based on the information, it is possible to intuitively specify the target. It is possible to display cases regardless of type, which also enables intuitive goal specification.

By displaying the modified image as well, it is possible to specify the result intuitively. Then, by selecting the area, the processing target can be limited and the effect of image modification can be enhanced. In addition, the degree of freedom in processing can be further increased by allowing registration as an additional case. Further, if a certain case is registered in the standard type, it can be used as a transfer type case from the information, and thus it is possible to accelerate the collection and accumulation of the transfer type case.

By displaying the evaluation index for image evaluation, it is possible to provide objective judgment material, so that the user can make an evaluation with more peace of mind and improve the skill level required for image modification. ..

Although the image processing apparatus, the image processing method, and the processing executed by the program of the present invention have been described in detail with reference to the embodiments shown in the drawings, the present invention is limited to the above-described embodiments. is not it. Therefore, other embodiments, additions, changes, deletions, etc. can be made within the range that can be conceived by those skilled in the art, and the present invention can be changed as long as the actions and effects of the present invention are exhibited in any of the embodiments. It is included in the range. Therefore, the present invention can also provide an image processing system including an image processing device and a recording medium on which the above program for being executed by a computer is recorded.

10 ... image processing device, 11 ... server device, 12 ... image reading device, 13 ... imaging device, 14 ... cable, 15 ... network, 20 ... input I / F, 21 ... CPU, 22 ... storage device, 23 ... auxiliary storage Device, 24 ... Output I / F, 25 ... Storage medium, 26 ... Drive, 27 ... Controller, 30 ... Control unit, 31 ... Image acquisition unit, 32 ... Case acquisition unit, 33 ... Information storage unit, 34 ... Case selection unit , 35 ... Case application unit, 36 ... Image output unit, 37 ... First format correction unit, 38 ... Second format correction unit, 40 ... Area selection unit, 41 ... Input image, 42 ... Toolbox, 43 ... Automatic selection tool , 50 ... Case registration department, 60 ... Image evaluation department

Japanese Unexamined Patent Publication No. 2007-221591 Japanese Unexamined Patent Publication No. 2004-129226 Japanese Patent No. 4646735 Japanese Unexamined Patent Publication No. 2008-294696 Japanese Patent No. 4910948 JP-A-2002-171408 Japanese Patent No. 4421761 Japanese Unexamined Patent Publication No. 2006-080746 Japanese Patent No. 4985243

Claims (7)

An image processing system that modifies images
An information storage unit that stores a plurality of case information including reference information at the time of modification that is referred to for specifying the content of the modification, and case identification information that can identify a case that has been modified according to the content of the modification.
An image acquisition unit that accepts image input and acquires the image,
A case selection unit that accepts selection of case information by arranging and displaying a plurality of case information stored in the information storage unit on the display unit based on the case identification information included in each case information.
A case application unit that modifies the image acquired by the image acquisition unit based on the selected case information, and a case application unit.
In order for the user to evaluate the quality of the image modified by the case application unit, the information of the modified image, or the information of the modified image and the image or target acquired by the image acquisition unit. An image evaluation unit that generates an evaluation index using the information of one image to be referred to and displays the evaluation index on the display unit is included.
The case selection unit displays one image referred to as a target for each case information, and accepts the designation of one target image.
The case application unit is an image processing system that modifies the designated image toward the one image. The case identification information includes a case identifier assigned to the case, a shooting position of the image used in the case, a shooting date and time, an image file name, color tone information, a scene name of a shooting scene, an object name to be shot, and the subject. The image processing system according to claim 1, further comprising an impression expression representing an impression of an image and at least one of the selected area information when selecting an area in the image. The image processing system according to claim 1 or 2, wherein the case selection unit displays the images modified by the case application unit side by side or alternately with the designated one image. As a region for modifying the image acquired by the image acquisition unit, at least one designation of an image space region, an image signal channel region, and an image signal frequency region is accepted, and at least one designated region is received. The image processing system according to any one of claims 1 to 3, further comprising an area selection unit for selecting. The modification performed on the image acquired by the image acquisition unit is used as an additional case, and the case registration unit for storing the case information obtained in the modification as additional case information in the information storage unit is further included. , The image processing system according to any one of claims 1 to 4. It is an image processing method executed by an image processing system that modifies an image, and the image processing system performs modification according to the modification reference information referred to for specifying the content of the modification and the content of the modification. Includes a case identification information that can identify a case, and an information storage unit that stores a plurality of case information including
The steps to accept the input of an image and acquire the image,
As a region for modifying the image acquired in the acquisition step, at least one designation of an image space region, an image signal channel region, and an image signal frequency domain is accepted, and at least one designated region is received. Steps to select and
A step of arranging and displaying a plurality of case information stored in the information storage unit on the display unit based on the case identification information included in each of the case information.
A step of accepting the selection of one of the plurality of case information displayed on the display unit, and
Based on the selected case information, the step of modifying the selected area of the image acquired in the acquisition step and the step of modifying the selected area.
The modification performed in the modification step is used as an additional case for the image acquired in the acquisition step, and the case information obtained in the modification is stored in the information storage unit as additional case information. When,
In order to allow the user to evaluate the quality of the image modified in the modification step, the information of the modified image, or the information of the modified image and the acquired image or the target is referred to. Including a step of generating an evaluation index using image information and displaying the evaluation index on a display unit.
In the step of accepting the selection, one image referred to as a target is displayed for each case information, and the designation of one target image is accepted.
In the retouching step, an image processing method for retouching the designated one image. A program for causing a computer to execute the image processing method according to claim 6.

Images