JP6797585B2 - Image processing device and its control method and program - Google Patents

Description

The present invention relates to an image processing apparatus that performs so-called rewriting processing that gives an arbitrary lighting effect to an image, and a control method and program thereof.

Conventionally, an image process that gives an effect of illuminating a subject in an image after shooting with light from a virtual light source (hereinafter referred to as a "virtual light source"), a so-called rewriting process, is known. This process is used, for example, to correct the shadow of a subject caused by ambient light in a more preferable manner when the shadow of the subject is not preferable.

In order to obtain a preferable shadow, a function of adjusting the position and direction of the virtual light source is indispensable, but a user interface that allows the user to intuitively adjust the position is desired. Patent Document 1 proposes displaying a light source as an icon in order to show the positional relationship of the light source with respect to the subject. The light source icon is depicted as directly illuminating the subject, while at the same time displaying a preview of the relighting effect on the subject.

Japanese Unexamined Patent Publication No. 05-210711

In the technique described in Patent Document 1, the icon of the virtual light source and the preview of the subject are expressed in the same space. Therefore, in order to display both the virtual light source and the subject preview, the farther the virtual light source is, the smaller the subject must be displayed. Also, if the virtual light source icon is moved to the front side of the subject, the icon hides the subject, making it difficult to see the preview.

It is an object of the present invention to eliminate such inconveniences and to present an image processing device capable of intuitively grasping a positional relationship such as a direction and a distance of a virtual light source with respect to a subject, and a control method and a program thereof.

In order to achieve the above object, the image processing apparatus according to the present invention has the direction of the virtual light source with respect to the designated subject and the distance of the virtual light source with respect to the designated subject among the images displayed on the display unit. The direction of the virtual light source with respect to the specified subject, including the changing means for changing the above, the image, and the first item corresponding to the specified subject and the second item corresponding to the virtual light source on the display unit. A display control means for controlling the display of a predetermined item indicating the above, a first operation of changing the position of the second item with respect to the first item in the predetermined item, and the designated subject. An image processing device having a receiving means for receiving a second operation for changing the distance of the virtual light source with respect to the above, in response to the receiving means receiving the first operation and the second operation. Te, wherein the the direction of the virtual light source with respect to the designated object, both the changing unit and the distance of the virtual light source with respect to the designated subject to.

According to the present invention, it is possible to realize a user interface that can intuitively grasp the positional relationship such as the direction and distance of the virtual light source with respect to the main subject.

It is a schematic block diagram of one Example of this invention. It is explanatory drawing of the example of a main image and a sub image. It is an operation flowchart of this Example. It is explanatory drawing of the 2nd example of a sub image. It is explanatory drawing of the 3rd example of a sub image. It is explanatory drawing of the 4th example of a sub image. It is explanatory drawing of the 5th example of a sub image. It is explanatory drawing of the 6th example of a sub image. It is explanatory drawing of a rewriting process.

Hereinafter, examples of the present invention will be described in detail with reference to the drawings.

FIG. 1 shows a schematic block diagram of an embodiment of an image processing apparatus according to the present invention.

Reference numeral 101 denotes a CPU that controls each module constituting the image processing device 100. Reference numeral 102 is a ROM for storing programs and parameters. The ROM 102 stores a control program executed on the CPU 101 for controlling the image processing device 100.

Reference numeral 103 denotes an operation unit that accepts operations by the user, specifically, changes and determinations of the position and distance of the virtual light source. The operation unit 103 includes buttons such as a mouse and a cross key, a touch panel, and the like. Here, a touch panel will be described as the operation unit 103.

Reference numeral 104 denotes a bus connecting each module. Reference numeral 105 denotes a DRAM that temporarily stores various data such as a main image and a sub image. The main image is an image that is the target of the rewriting process. The sub-image is an operation image or an auxiliary image for indicating the position and direction of the virtual light source with respect to the main subject included in the main image and manipulating the position and direction of the virtual light source during the rewriting process. Reference numeral 106 denotes a memory I / F that mediates access to the DRAM 105 by each module.

Reference numeral 108 denotes a main subject detecting means for detecting or determining a main subject to be relighted from one or a plurality of subjects included in the main image. Examples of the method of identifying the main subject include a method of identifying the largest face by face detection and a method of detecting the subject in a range designated by the operation unit 103 by the user and setting the main subject as the main subject.

Reference numeral 109 denotes a rewriting means for performing rewriting image processing that creates a shadow on the main subject by irradiating the main subject included in the main image with the output light of the virtual light source. As a premise of the rewriting process, it is necessary to know the three-dimensional shape of the target subject, and the three-dimensional shape is measured at the time of imaging and input to the image processing device 100 together with the captured image. The three-dimensional shape is determined, for example, from the distance information acquired for each pixel regarding the distance between the camera and the subject at the time of shooting. Another method is to use a stereo camera to obtain a distance image.

The basics of the rewriting process will be described with reference to FIG. The three-dimensional shape of the subject 900 is estimated, and the normal vectors 901 to 907 are calculated at predetermined intervals. 908 is an ambient light vector, and 909 and 910 are virtual light source vectors. In the absence of the virtual light source (virtual light source vectors 909 and 910), the closer the directional components of the normal vectors 901 to 907 and the ambient light vector 908 are to 180 degrees, the brighter the subject 900 is. Specifically, the region around the normal vector 906 is brighter than that around the normal vector 901. When the rewriting process is performed using the virtual light source vector 909, the area around the virtual light source vector 909 and the normal vector 905-906 whose direction component is close to 180 degrees becomes bright. When the rewriting process is performed using the virtual light source vector 910, the area around the virtual light source vector 910 and the normal vector 901 to 902 whose directional components are close to 180 degrees becomes bright.

Reference numeral 110 denotes a sub-image creating means (auxiliary image creating means) for creating a sub-image (an auxiliary image for manipulating the positional / distance relationship between the main subject and the virtual light source included in the main image). The detailed processing of the sub-image creating means 110 will be described later.

Reference numeral 111 denotes a display unit that displays the main image rewritten by the rewriting means 109 and the sub image created by the sub image creating means 110. The display unit 111 may display the main image and the sub image on the same screen at the same time, or may display them on separate screens. In the latter case, the display unit 111 has two display devices or has two display windows on the window display system of the computer system.

Reference numeral 112 denotes a display interface (I / F) that outputs a main image and a sub image to the display unit 111.

Reference numeral 120 denotes a recording medium for storing image data of a main image including a main subject to be rewritten. Three-dimensional shape data of the main subject is also recorded on the recording medium 120. Instead of the recording medium 120, it may be a server on the network that provides the main image and the three-dimensional shape data of the main subject to the outside. Further, the image data and the three-dimensional shape data of the main subject may be supplied from the network camera and the distance image camera that measures the three-dimensional shape of the main subject.

A method of changing the position and distance of the main image, the sub-image created by the sub-image creating means 110, and the virtual light source will be described with reference to FIG. FIG. 2A shows the main image 200, in which a person 201, which is the main subject, is shown. 2 (b) to 2 (e) show the sub-image 202. 2 (b) and 2 (c) are views for explaining an operation method for changing the position of the virtual light source, and show how the position of the virtual light source is changed by a slide operation. 2 (d) and 2 (e) are diagrams for explaining an operation method for changing the distance from the main subject to the virtual light source, and show how the distance is changed by a pinch operation.

In FIGS. 2 (b) to 2 (e), 203 is a subject object indicating a position corresponding to the person 201. In this embodiment, a humanoid pattern is used as the subject object 203.

Reference numeral 204 denotes a virtual light source object indicating the position of the virtual light source for performing rewriting image processing on the person 201. In this embodiment, the design of the flashlight is used as the virtual light source object 204, and the light of the flashlight is drawn so as to face the subject object 203.

Reference numeral 205 denotes a spherical surface having the subject object 203 at the center and having a radius corresponding to the distance from the subject object 203 to the virtual light source object 204. A graticule on the front side of the subject object 203 is drawn on the spherical surface 205.

Reference numeral 206 denotes a user's hand trying to change the position of the virtual light source object 204 by a slide operation using the operation unit 103 (touch panel). Here, the user changes the position of the virtual light source object 204 by sliding the operation unit 103 while referring to the position of the virtual light source object 204 displayed in the sub image. 207 indicates a user's hand that changes the distance of the virtual light source object 204 to the main subject by a pinch operation of the operation unit 103. As described above, in this embodiment, the position or the illumination direction of the virtual light source with respect to the main subject is adjusted by the slide operation on the virtual light source object 204, and the distance to the main subject is adjusted by the pinch-in / out operation.

In the example shown in FIG. 2, as shown in FIG. 2B, the hand 206 slides the virtual light source object 204 with respect to the subject object 203 from the lower right to the upper right in the operation unit 103 to position the virtual light source. To change. Before moving the virtual light source, the sub-image creating means 110 draws the flashlight so that the light of the flashlight faces the upper left, which is the direction of the subject object 203. The sub-image creating means 110 draws the virtual light source object 204 so that the flashlight of the virtual light source object 204 gradually changes its direction toward the subject object 203 even while the virtual light source is moving. When the position of the virtual light source object 204 is determined by releasing the hand 206 from the operation unit 103 at the upper right position with respect to the subject, the sub-image creating means 110 turns the flashlight toward the lower left and sets the virtual light source object 204. Stop drawing.

As shown in FIG. 2C, the position of the virtual light source can be changed from the front side to the back side by sliding the hand 206 so as to move from the range of the spherical surface 205 to the outside of the range of the spherical surface 205 in the operation unit 103. .. The sub-image creating means 110 draws the virtual light source object 204 behind the graticule of the spherical surface 205 in a smaller size than when it is on the front side. By such drawing, it is expressed that the virtual light source is located on the back side. On the contrary, when the hand 206 slides from outside the range of the spherical surface 205 into the range, the virtual light source is moved from the back to the front side.

A virtual pinch operation is performed from the state in which the two fingers of the hand 207 are spread as shown in FIG. 2 (d) to the state in which the two fingers of the hand 207 are narrowed as shown in FIG. 2 (e). Bring the light source closer to the subject. During the pinch operation, the sub-image creating means 110 draws the spherical surface 205 so as to gradually become smaller, and moves the virtual light source object 204 attached to the spherical surface 205 in accordance with the change in the size of the spherical surface 205. , Draw the virtual light source object 204.

When the distance becomes the farthest infinity or the closest predetermined distance, the sub-image creating means 110 is characterized by drawing the spherical surface 205, such as blinking the spherical surface 205 or changing its color. Is added to notify the user.

FIG. 3 shows an operation flowchart of this embodiment. The characteristic operation of this embodiment will be described with reference to FIG. A program that realizes the operation shown in FIG. 3 is stored in the ROM 102. The CPU 101 realizes the operation shown in FIG. 3 by reading and executing the program from the ROM 102.

In S301, the CPU 101 sets default values for the position information and the distance information of the virtual light source, and outputs them to the rewriting means 109 and the sub-image creating means 110. The default value may be a fixed value such as directly in front of the subject, or may be set to a recommended position by determining the shadow due to the ambient light.

In S302, the CPU 101 causes the main subject detecting means 108 to identify the main subject. The main subject detecting means 108 detects the main subject from the main image 200 read from the DRAM 105 via the memory I / F 106. In the example shown in FIG. 2, the main subject detecting means 108 identifies the person 201 as the main subject.

In S303, the CPU 101 instructs the rewriting means 109 to execute the rewriting process by the virtual light source. The rewriting means 109 sets the position of the virtual light source based on the position information and the distance information of the virtual light source supplied from the CPU 101. Then, the rewriting means 109 reads the main image 200 from the DRAM 105 via the memory I / F 106, and performs rewriting image processing on the person 201 of the main image 200. The rewriting means 109 stores the rewritten main image in the DRAM 105 via the memory I / F 106.

In S304, the CPU 101 causes the sub-image creating means 110 to generate a sub-image. The sub-image creating means 110 creates a sub-image by drawing the virtual light source object 204 and the spherical surface 205 based on the position information and the distance information of the virtual light source from the CPU 101. Then, the sub-image creating means 110 stores the created sub-image in the DRAM 105 via the memory I / F 106.

In S305, the CPU 101 displays the main image and the sub image on the display unit 111. That is, the CPU 101 reads the rewritten main image 200 and the sub image 202 from the DRAM 105 via the memory I / F 106, and supplies the rewritten main image 200 and the sub image 202 to the display unit 111 via the display I / F 112.

In S306, the CPU 101 determines whether or not the user has input an instruction to change the position or distance of the virtual light source from the operation unit 103. If the position or distance of the virtual light source is changed, the CPU 101 proceeds to S308, and if it is not changed, the CPU 101 proceeds to S307.

In S307, the CPU 101 determines whether or not an instruction for determining the position and distance of the virtual light source has been input from the operation unit 103 by the user. When the position and distance of the virtual light source are determined, the CPU 101 ends the process shown in FIG. 3, and returns to S306 when the position or distance is not determined.

In S308, the CPU 101 acquires the position information and the distance information of the virtual light source based on the position and distance of the virtual light source changed from the operation unit 103 by the user, and outputs the position information and the distance information to the rewriting means 109 and the sub-image creating means 110. After that, the CPU 101 returns to S303 and recreates the main image and the sub image based on the changed position / distance of the virtual light source.

The CPU 101 records the rewriting processing result obtained by the processing shown in FIG. 3 on the recording medium 120, or transmits it to the outside via a network (not shown).

FIG. 4 shows another example of the sub image. In FIG. 2, the direction of the virtual light source (illumination direction) with respect to the main subject is expressed by the direction of the flashlight, but in FIG. I made it. That is, in the sub-image 402 shown in FIG. 4, a shadow is drawn on the subject object 403 according to the direction of the virtual light source object 404 with respect to the main subject.

FIG. 4A shows a sub-image 402 in which a virtual light source is applied to the main subject from the lower right on the front side. FIG. 4B shows a sub-image 402 in which a virtual light source is applied to the main subject from the upper right on the back side. FIG. 4C shows a sub-image 402 in which a virtual light source is applied to the main subject from the right side. As shown in FIG. 4, a shadow formed on the subject object 403 when illuminated from the direction of the virtual light source object 404 is drawn in the same manner as the ebb and flow of the moon.

FIG. 5 shows yet another configuration of the sub-image. In the sub-image 502 shown in FIG. 5, a shadow 506 cast by the main subject (subject object 503) on the spherical surface 505 is drawn by the virtual light source object 504. The position of the shadow 506 makes it easier for the user to recognize the direction of the virtual light source with respect to the main subject.

FIG. 5A shows a sub-image 502 in a state where the main subject is irradiated with a virtual light source from the lower right on the front side. At this time, since the shadow 506 is formed on the back side of the spherical surface 505, it is drawn in accordance with the spherical surface 505 below the graticule line of the spherical surface 505. When the color of the graticule line on the front side is close to that of the shadow 506, the color of the graticule line on the front side in the area of the shadow 506 may be inverted and drawn in order to improve visibility.

FIG. 5B shows a sub-image 502 in a state where the main subject is irradiated with a virtual light source from the upper right on the back side. At this time, since the shadow 506 is formed on the front side of the spherical surface 505, the shadow 506 is drawn so as to erase the graticule of the spherical surface 505 in the region of the shadow 506. Further, since the shadow 506 in FIG. 5 (b) is located on the front side of that in FIG. 5 (a), the shadow 506 is drawn so that the shadow 506 in FIG. 5 (b) is larger than that in FIG. 5 (a). To do.

FIG. 5C shows a sub-image 502 in a state where the main subject is irradiated with a virtual light source from the right side on the front side. At this time, since the shadow 506 can be formed on both the front side and the back side of the spherical surface 505, the part that falls on the back side of the shadow 506 is drawn below the graticule line, and the part that falls on the front side erases the graticule line. Is drawn as.

FIG. 5D shows a sub-image 502 when the distance of the virtual light source from the main subject is closer than that of FIG. 5A. At this time, the shadow 506 with respect to the spherical surface 505 is drawn larger in FIG. 5 (d) than in FIG. 5 (a). The reason for this will be described with reference to FIGS. 5 (e) and 5 (f). FIG. 5E shows a case where the distance from the main subject to the virtual light source is infinity. In this case, since the directions of the light rays of the virtual light source are parallel, the shadow cast by the subject is the same size as the subject. On the other hand, FIG. 5 (f) shows a case where the distance from the main subject to the virtual light source is short. In this case, since the direction of the light rays of the virtual light source is radial, the shadow cast by the subject becomes larger than the size of the subject.

The sub-image shown in FIG. 5 may also be used in combination with the display method shown in FIG. That is, the shadow 506 is displayed on the spherical surface 505 while the illuminated portion is displayed brightly (the non-illuminated portion is darkened) on the subject object.

FIG. 6 shows a fourth example of the sub-image. In the sub-image 602 shown in FIG. 6, the closer the virtual light source is to the main subject, the larger the subject object is drawn, and the farther the virtual light source is, the smaller the subject object is drawn.

FIG. 6A shows a sub-image 602 when the distance of the virtual light source to the main subject is short. FIG. 6B shows a sub-image 602 when the virtual light source is far away from the main subject. FIG. 6C shows a sub-image 602 when the distance of the virtual light source to the main subject is infinity.

As shown in FIG. 6, the closer the virtual light source object 604 is to the subject object 603, the larger the subject object 603 is drawn. As shown in FIG. 6C, when the virtual light source object 604 is at an infinity distance from the subject object 603, the subject object 603 may be drawn in the shape of a point.

FIG. 7 shows a fifth example of the sub-image. In the sub-image 702 shown in FIG. 7, a fluorescent lamp type virtual light source object 704 is adopted as in the virtual light source object 204 shown in FIG. 2, and the size of the virtual light source object 704 is set according to the distance to the main subject. I changed it.

FIG. 7A shows a sub-image 702 in a state where the virtual light source is close to the main subject and is illuminated from the lower right on the front side. FIG. 7B shows a sub-image 702 in a state where the virtual light source is far from the main subject and is illuminated from the lower right on the front side.

As shown in FIG. 7, the closer the virtual light source object 704 is to the subject object 703, the larger the virtual light source object 704 is drawn.

FIG. 8 shows a sixth example of the sub-image. In the sub-image 802 shown in FIG. 8, the distance between the main subject and the virtual light source is easily recognized by clearly indicating the distance between the subject object 803 and the virtual light source object 804. Reference numeral 807 is an arrow connecting the subject object 803 and the virtual light source object 804. Reference numeral 808 is a balloon indicating the distance numerically.

FIG. 8A shows a sub-image 802 in a state where the distance of the virtual light source to the main subject is 2.5 m and irradiation is performed from the lower right on the front side. The balloon 808 displays 2.5 m.

FIG. 8B shows a sub-image 802 in a state where the distance of the virtual light source is infinity with respect to the main subject and irradiation is performed from the upper right at the boundary between the front side and the back side. A symbol ∞ indicating infinity is displayed in the balloon 808.

In FIG. 1, the main subject detecting means 108, the rewriting means 109, and the sub-image creating means 110 are described as components separate from the CPU 101 so that the functions can be easily understood. However, some or all of the functions of the main subject detecting means 108, the rewriting means 109, and the sub-image creating means 110 can be realized by a program that operates on the CPU 101. Further, some of the functions of the main subject detecting means 108, the rewriting means 109, and the sub-image creating means 110 may be realized by a dedicated logical operation circuit.

An object of the present invention is also achieved by supplying the device with a storage medium in which a program code of software that realizes the functions of the above-described embodiment is recorded. At this time, the computer (or CPU or MPU) including the control unit of the supplied device reads and executes the program code stored in the storage medium. The program code itself read from the storage medium realizes the functions of the above-described embodiment, and the program code itself and the storage medium storing the program code constitute the present invention.

As the storage medium for supplying the program code, for example, a magnetic disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a non-volatile memory card or a ROM, or the like can be used.

In some cases, the OS (basic system or operating system) running on the device performs part or all of the processing based on the instructions in the above program code, and the processing realizes the functions of the above-described embodiment. included.

Further, the case where the program code read from the storage medium is written in the memory provided in the function expansion board inserted in the device or the function expansion unit connected to the computer to realize the function of the above-described embodiment is also included. Is done. At this time, based on the instruction of the program code, the function expansion board, the CPU provided in the function expansion unit, or the like performs a part or all of the actual processing.

Although preferable examples of the present invention have been described above, the present invention is not limited to these examples, and various modifications and modifications can be made within the scope of the gist thereof.

The present invention supplies a program that realizes one or more functions of the above-described embodiment to a system or device via a network or storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by the processing to be performed. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Claims (11)

Among the images displayed on the display unit, a changing means for changing the direction of the virtual light source with respect to the specified subject and the distance of the virtual light source with respect to the specified subject.
A predetermined item indicating the direction of the virtual light source with respect to the specified subject, including the first item corresponding to the designated subject and the second item corresponding to the virtual light source, is displayed on the display unit together with the image. Display control means to control
A reception that accepts a first operation of changing the position of the second item with respect to the first item and a second operation of changing the distance of the virtual light source to the designated subject in the predetermined item. An image processing device having means and
In response to the reception means receiving the first operation and the second operation, both the direction of the virtual light source with respect to the designated subject and the distance of the virtual light source with respect to the designated subject are both set. An image processing apparatus characterized in that the changing means changes. A detection means for detecting the main subject included in the image, and
When displaying a screen for changing the direction of the virtual light source by the changing means on the display unit, the virtual light source is set to be in a predetermined direction with respect to the main subject detected by the detecting means. The image processing apparatus according to claim 1, further comprising a display control means for controlling the display of the screen in the state of the above. The image processing apparatus according to claim 2, wherein the predetermined direction is a direction directly in front of the main subject. A circle centered on the first item is drawn in the predetermined item, and the direction can be changed by moving the second item in the circle. The image processing apparatus according to claim 1 or 2. Further having a touch detecting means capable of detecting a touch operation on the display unit,
The image processing device according to claim 4, wherein the second item can be moved by a touch operation on the display unit. The image processing device according to claim 4 or 5, wherein the second item can be moved by a touch slide operation on the display unit. The method according to any one of claims 1 to 6, wherein when the direction is changed by the changing means, the image displayed on the display unit is recreated based on the changed direction. Image processing equipment. The invention according to any one of claims 1 to 7, wherein the designated subject is controlled to execute the rewriting process based on the direction of the virtual light source and the distance of the virtual light source. Image processing equipment. Among the images displayed on the display unit, a change step for changing the direction of the virtual light source with respect to the specified subject and the distance of the virtual light source with respect to the specified subject, and
A predetermined item indicating the direction of the virtual light source with respect to the specified subject, including the first item corresponding to the designated subject and the second item corresponding to the virtual light source, is displayed on the display unit together with the image. Display control steps to control
Accepting a first operation of changing the position of the second item with respect to the first item and a second operation of changing the distance of the virtual light source to the designated subject in the predetermined item. An image processing device having steps and
In response to receiving the first operation and the second operation in the reception step, both the direction of the virtual light source with respect to the designated subject and the distance of the virtual light source with respect to the designated subject are both set. A control method for an image processing apparatus, which comprises changing in the change step . A program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 8. A computer-readable recording medium containing a program for causing the computer to function as each means of the image processing apparatus according to any one of claims 1 to 8.

Images