JP7657602B2 - IMAGE SYNTHESIS METHOD, COMPUTER PROGRAM, AND IMAGE SYNTHESIS DEVICE - Google Patents

Description

The present invention relates to an image synthesis method, etc.

Conventionally, there is a technique for compositing and arranging an object in a captured image.
Patent Document 1 proposes a method of generating a real coordinate space of the shooting space by acquiring the distance to the subject when shooting an image, and processing and displaying an object according to the specified real space coordinates.

Patent No. 6685814

In the technology described in Patent Document 1, an object image is processed and synthesized according to the coordinates of a specified synthesis position in a real coordinate space generated from a captured image and distance information. The synthesis position is specified on a two-dimensional image in which three-dimensional coordinates are projected onto a two-dimensional captured image, and the Z coordinate (depth position) is determined at the time the XY coordinates (planar position) are specified. Therefore, it is not possible to synthesize an object image by arbitrarily changing the depth position (Z coordinate) while keeping the planar position (XY coordinates) on the captured image fixed.

In view of the above problems, the present invention aims to provide an image synthesis method that solves the above problems and allows intuitive operation to change the synthesis position and depth distance on a plane.

In order to achieve the above object, the image synthesis method of the present invention comprises the steps of:
1. A method for compositing a second image onto a first image, comprising the steps of:
a position detection step of detecting a designated position on the first image for synthesizing the second image;
a composite display step of changing a size of the subject of the second image based on subject distance information of the first image at the specified position and size information of the subject of the second image so that the size corresponds to the subject distance of the first image at the specified position, and displaying the composite image;
a depth distance change detection step of detecting a depth distance change operation for disposing the second image at a depth distance different from the subject distance at the specified position on the specified position;
a size changing step of changing a size of the second image in response to the depth distance change operation;
a depth distance information display step of displaying changed depth distance information in response to the depth distance change operation;
having
In the depth distance information display step, a region of the first image corresponding to the depth distance is highlighted .

The present invention provides an image synthesis method that allows intuitive changes to the synthesis position and depth distance on a plane.

1 is a block diagram showing a main functional configuration for executing an image synthesis method according to an embodiment of the present invention; 4 is a flowchart illustrating a process executed in the embodiment. FIG. 11 is a diagram showing a display method relating to a first image selection executed in the embodiment. FIG. 11 is a diagram showing a display method relating to a second image selection executed in the embodiment. 13A to 13C are diagrams illustrating subject extraction according to the second image selection executed in the embodiment. FIG. 13 is a diagram for explaining the position of a reference point on a second image in the embodiment. 11A and 11B are diagrams illustrating a method for changing the size of a second image in accordance with a subject distance of a first image and synthesizing the first image in an embodiment. 13A to 13C are diagrams illustrating an example of a change in size of a second image to be synthesized when the depth distance is changed, and a method of displaying the depth distance to be synthesized. 13A and 13B are diagrams illustrating another method of displaying the depth distance synthesized in the embodiment.

Below, preferred embodiments of the present invention will be described using examples with reference to the attached drawings. Note that in each drawing, the same members or elements are given the same reference numbers, and duplicate descriptions will be omitted or simplified.

In the embodiment described below, an example of executing the image synthesis method of this embodiment using a computer as an image processing device will be described. That is, this embodiment can be executed on a desktop computer, a laptop computer, a tablet computer, a smartphone, etc.

The image synthesis method of this embodiment acquires a captured image and distance information (distance map information) corresponding to the captured image from an imaging device such as a digital camera, and enables intuitive synthesis of another image at a desired planar position and depth position (depth distance) of the captured image. In addition, at that time, an image of an appropriate size according to the depth position (depth distance) is synthesized.

The configuration of a computer as an image processing device that executes the image synthesis method of this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the main functional configuration for executing the image synthesis method of this embodiment, and does not show peripheral components.
1 has a user interface for receiving operational input from a user, such as a mouse, a keyboard, a joystick, a touch panel display, etc. The input unit 10 also has an interface (I/F) for receiving images and distance information (distance map information) from a digital camera or other external device.

The control unit 11 controls data in the input unit 10, the calculation unit 12, the storage unit 13, and the output unit 14, and controls each unit of the entire image processing device 100. That is, the control unit 11 has a built-in CPU as a computer, and controls each unit of the entire image processing device 100 via a bus based on a computer program stored in a non-volatile memory.
The calculation unit 12 has a calculation processing device and executes a calculation processing program stored in the storage unit 13, for example.

The storage unit 13 is composed of a primary storage device such as a semiconductor memory and a secondary storage device such as a hard disk. The storage unit 13 stores a calculation processing program as a computer program, and also stores various information such as image data, distance information, subject size information, and parameters input via the input unit 10. The storage unit 13 also stores the image synthesized by the calculation unit 12.

The output unit 14 includes a display as a display unit and an I/F for outputting images and the like to the outside. In this embodiment, the display of the output unit 14 is, for example, a touch panel display that also has a user interface function.

Next, FIG. 2 is a flow chart for explaining the processing executed in the embodiment, and the image synthesis processing of the embodiment will be explained with reference to the flow chart of FIG.
The processing shown in the flowchart of FIG. 2 is realized by the control unit 11 reading out a computer program stored in, for example, non-volatile memory in the storage unit 13, expanding it into volatile memory in the control unit 11, and executing it, thereby causing each unit to operate.

In step S201, the control unit 11 displays a plurality of images stored in the storage unit 13 as candidates for the first image (background image) on the touch panel display of the output unit 14. Note that Fig. 3 is a diagram showing a display method related to the first image selection executed in this embodiment, and the display is a thumbnail display as shown in Fig. 3, or a single image is displayed so that the displayed image can be changed by a user operation.
When the user performs a tap operation on the display area of a desired image on the displayed images via the input unit 10, a first image (background image) is selected.

In this embodiment, the image stored in the memory unit 13 is an image to which subject distance information (subject distance map information) has been added, and is an image input to the image processing device 100 from an imaging device (camera) and a distance measurement unit (not shown) via the interface of the input unit 10. Alternatively, it is an image to which subject distance information (subject distance map information) has been added, which has been captured by an external imaging device, and which is temporarily saved in an external device, and then acquired via the interface of the input unit 10 and stored in the memory unit 13. Here, the subject distance information (subject distance map information) is, for example, information on the subject distance recorded for each predetermined pixel.

In step S202, the control unit 11 displays the first image (background image) selected in step S201 in a part (area 401) of the touch panel display of the output unit 14 as shown in FIG.
FIG. 4 is a diagram showing a display method relating to a second image selection executed in the embodiment.
In an area 402 of the touch panel display different from an area 401 in which the first image is displayed, a plurality of candidate images to be combined with the first image are displayed so that the user can select them. Note that although thumbnail display is shown as an example in FIG. 4, other display methods may be used.

Here, we have described the order of selecting a first image in step S201, and then displaying the first image and multiple images to be combined in step S202, but this order is not necessarily the only option. Multiple images to be combined can also be displayed in advance in area 402.

A configuration may also be adopted in which a plurality of first image candidates are stored in a specific storage area, one of which is displayed in area 401, and which allows the user to change to another first image by swiping area 401. Alternatively, a configuration may also be adopted in which a left-pointing arrow and a right-pointing arrow are displayed on the left and right periphery of area 401, respectively, and the other images in the storage area can be displayed in sequence by tapping on either arrow.

With this configuration, it is possible to easily change the first image used as the background image without using complex menu selection or the like.
In step S203, the user operates the touch panel display of the input unit 10 and selects an image to be combined with the first image (background image) from the area 402 by tapping or touching. The selected image to be combined is set as the second image. That is, the image displayed at the coordinates on the touch panel display touched by the user's finger becomes the selected second image.

The control unit 11 acquires the selected second image together with meta information associated with the second image. The meta information is saved, for example, as tag data saved in Exif format or maker note information. The meta information acquired in step S203 includes information related to the size of the subject present in the second image (size information), etc.

The subject size information includes area information (coordinate information) of the subject recognized in the second image and information on its size. The subject size information may also include information on how many millimeters one pixel of the image corresponds to in object space, distance information to the object (subject) recognized by face recognition or object recognition, focal length information when the subject was photographed, or other information that allows the size of the subject to be calculated. Alternatively, it may include information on a map of layers when the image is divided into layers according to the distance to the subject.

In the case of an image to which object recognition information or distance layer information is attached as meta information, it is also possible to extract a specific subject or a specific layer from the image displayed at the coordinates on the touch panel display touched by the user's finger and use it as a second image.
5A and 5B are diagrams for explaining subject extraction related to the second image selection executed in the embodiment. For example, as shown in Fig. 5A, two people 502 and 503 are shown in an image 501, and when the area of the person 502 is touched and selected, only the subject area is cut out as shown in Fig. 5B as 502' and used as the second image. In this way, the second image may be a specified subject image extracted from a specified image.

When the second image is selected, the control unit 11 sets a point in the selected second image that will be used as a reference for image synthesis.
6A and 6B are diagrams for explaining the position of a reference point on a second image according to an embodiment. For example, when the second image is rectangular, the reference point is set to any one of the vertices of the upper right 601, lower right 602, upper left 603, and lower left 604, any one of the midpoints of each side (605 to 608), or the center point 609 of the second image, as shown in Fig. 6A. Alternatively, the reference point may be set to the coordinates 610 of the contact position on the second image when the user selects the second image by a touch operation.

In particular, since the subject is often in contact with the ground or the like, it is preferable to set the reference point at the midpoint 607 of the lower side.
If the second image is the shape of a specific subject in the selected image, one of the topmost end 611, bottommost end 612, rightmost end 613, or leftmost end 614 of the specific subject is selected, as shown in FIG. 6B.

Alternatively, the reference point is set to any of the topmost end 615 of the midpoint of the shape width of the specific subject, the bottommost end 616 of the midpoint of the shape width, the rightmost end 617 of the midpoint of the shape height, the leftmost end 618 of the midpoint of the shape height, and the midpoints 619 of both the shape height and the shape width. It is particularly preferable to set the reference point to the bottommost end 616 of the midpoint of the shape width. Thus, step S203 includes a reference selection step of selecting a reference position in the second image while displaying the reference position of the second image.

In step S204, the user drags the second image selected by touch operation in step S203 on the touch panel display, for example, and drops it by releasing the finger on the first image displayed on the touch panel display. In this case, the position of the finger on the dropped first image (background image), i.e., the position where the finger is released, is set as the synthesis position (designated position) on the plane where the second image is synthesized.

Here, step S204 functions as a position detection process (position detection means) that detects a designated position on the first image for synthesizing the second image, and detects the position where the user drops the second image after dragging it on the touch panel display.
The control unit 11 acquires coordinates (composite position coordinates) on the first image (background image) that correspond to the coordinates of the composite position on the touch panel display. That is, the control unit 11 acquires distance information of the subject in the first image that corresponds to the composite position coordinates from the meta information of the first image.

Incidentally, instead of dragging and dropping, in step S203, the second image may be selected, for example, by a tap operation, and then the finger may be released, and the combination position may be specified by touching (tapping) the touch panel display on which the first image is displayed.
In addition, when specifying the synthesis position, if the position touched (tapped) on the touch panel display is set as the synthesis position, it may be difficult to finely adjust the synthesis position. Therefore, in order to clearly identify the synthesis position, a position that does not overlap with the user's finger in an area shifted from the contact position may be highlighted as the synthesis position.

In step S205, the control unit 11 transmits information on the subject distance of the first image at the composite position coordinates on the acquired first image (background image) and information on the size of the second image to the calculation unit 12. Based on the input information, the calculation unit 12 calculates size information when the second image is placed at the subject distance of the composite position coordinates on the first image.

That is, based on the subject distance D1 at the synthesis position coordinates on the first image, the focal length f1 when the first image was captured, and the size of one pixel in the first image (image side pixel size) s1, S1 is calculated using the following equation 1.
Here, S1 is the size of one pixel in the first image at the distance of the subject in the synthesis position coordinates (object-side pixel size).
S1=D1s1/f1 (1)

In addition, based on the size information of the subject in the second image, a converted pixel number PN1 is calculated by converting the size Os of the subject in the second image (image to be synthesized) into how many pixels it corresponds to using the object-side pixel size S1.
Next, in step S206, the calculation unit 12 calculates a scaling ratio R of the second image using the ratio between the number of pixels PN2 of the object in the second image and the number of converted pixels PN1 calculated in step S205, and enlarges/reduces the second image based on the calculated scaling ratio R. Here, the reference position for image synthesis in the second image set in step S203 is reset to match the enlarged/reduced second image.

In step S207, the control unit 11 performs synthesis so that the reference point of the second image after enlargement/reduction calculated in step S206 coincides with the synthesis position coordinates on the first image specified in step S205. That is, synthesis is performed by matching the reference position selected in step S203 with the specified position on the first image.
The compositing is performed by replacing the first image with the scaled second image in the area where the scaled second image is to be composited.

However, in the area of the first image into which the second image is composited, areas that are at a subject distance in front of the composite position of the second image display the first image as is without being replaced with the second image. In other words, in the area of the first image into which the second image is composited, areas that are at a depth distance from the specified position display the second image, and areas that are in front of the depth distance from the specified position display the first image. This enables composition that takes into account the depth relationship in front of and behind the subject.

By this processing, when the subject of the second image is placed at a depth distance from the imaging device (camera) to the subject in the composition position coordinates on the first image, the size of the subject of the second image is changed to the size observed from the imaging device and composited. That is, when the subject of the second image is composited, if the depth distance (subject distance) of the first image at the composition position is close, the subject of the second image is composited at a large size on the first image. On the other hand, if the depth distance (subject distance) of the first image at the composition position is far, the subject of the second image is composited at a small size.

In this way, step S207 functions as a composite display step (composite display means). Also, based on subject distance information of the first image at the specified position and subject size information of the second image, the size of the subject of the second image is changed to a size that corresponds to the subject distance of the first image at the specified position, and then the composite display is performed.

FIG. 7 is a diagram for explaining a method of synthesizing a first image by changing the size of the second image in accordance with the subject distance of the first image in this embodiment.
As shown in Fig. 7, the selected second image 701 is resized and synthesized according to the depth distance (subject distance) of a synthesis position 702 on the first image. 703 in Fig. 7 is the second image 701 that has been resized and synthesized.

The processes from step S204 to step S207 may be executed after the user completes the drag-and-drop operation, or the processes from step S204 to step S207 may be executed periodically while the user is performing the drag operation. In other words, the position detection process and the composite display process may be executed periodically while the user is moving the specified position.

By executing this periodically, the size of the second image can be changed according to the distance on the first image, allowing the user to change the drop position while checking the size of the second image. In addition, since the composite position on the first image and the reference point of the second image are always displayed to coincide, it is possible to determine the placement (composite position) while recognizing these positions.

Next, in step S208, the control unit 11 determines whether or not to change the depth distance (depth position, subject distance) at which the second image arranged in step S207 is synthesized in the first image.
When determining whether to change the depth distance, the synthesis position is determined, for example, when the user drags the second image onto the first image and releases his/her finger at the desired synthesis position coordinates.

At that time, if it is detected that the finger has been released after touching the synthesis position coordinates for longer than the predetermined time, or that the finger has been released after pressing the synthesis position coordinates with a pressure of a predetermined value or more, it is determined that the depth distance (subject distance) is to be changed. Alternatively, it may be determined that the depth distance is to be changed if the second image is dragged to the desired synthesis position coordinates and kept touching, and another finger is then touched on the display, that is, if two or more touches are detected.

The method for determining that the depth distance is to be changed is not limited to the above method, and other methods may be used. Here, step S208 functions as a depth distance change detection process (depth distance change detection means) that detects a depth distance change operation for placing the second image at a specified position at a depth distance different from the subject distance at the specified position. As described above, in this embodiment, the depth distance change detection process detects that a depth distance change operation has been performed based on the time or pressure with which the user touches the touch panel display, or by touching multiple points.

If it is determined in step S208 that the depth distance for placement is to be changed, the process proceeds to step S209, and if it is determined that the depth distance is not to be changed, the process proceeds to step S210.
In step S209, the control unit 11 changes the depth distance to be placed at the synthesis position coordinates, for example, depending on the touch time or touch pressure, and changes the size of the second image depending on the changed depth distance, and displays it synthesized on the first synthesis position coordinates.

That is, step S209 functions as a size change process (size change means) that changes the size of the second image in response to the depth distance change operation. It also functions as a depth distance information display process (depth distance information display means) that displays the changed depth distance information in response to the depth distance change operation. In this embodiment, in the depth distance change detection process, the depth distance is changed in response to the time or pressure with which the user touches the touch panel display or a pinch operation.

The change in the depth distance may be set so as to gradually change the distance away from the object, or so as to gradually change the distance toward the object. Note that the direction of change may be selectively set in advance.
Furthermore, the size of the second image to be combined with the depth distance may be changed at a predetermined touch time interval and displayed, and when the set maximum or minimum distance is reached, the depth distance may be gradually changed toward the minimum or maximum distance.

The depth distance may be changed according to the touch pressure as well as the touch time. The intervals at which the depth distance is changed may be equal or may vary according to the distance. In this case, it is preferable to increase the intervals at which the depth distance is changed as the distance increases.

If the distance is changed when two or more touch points are detected in step S208, a pinch operation (an operation to change the distance between two touch points) may be detected and the depth distance may be changed in accordance with the pinch operation. In a pinch operation, for example, when the distance between two touch points is increased, the depth distance is made closer, and when the distance between two touch points is decreased, the depth distance is made farther. In this case, the size of only the second image, which is a specific subject, is changed, and the corresponding depth distance is displayed on the screen, for example, as a number, making it easier to synthesize three-dimensional positions, unlike conventional zooming in and out using a pinch operation.

In addition, it is preferable to highlight the subject in the first image that is at the same depth distance as the changed depth distance so that the user can visually recognize the changed depth distance. In addition, in this case, it is preferable to highlight the image area of the corresponding depth distance in a specific color or with a different brightness.

FIG. 8 is a diagram for explaining a change in size of the second image to be synthesized when the depth distance is changed, and an example of a method of displaying the depth distance to be synthesized.
For example, as shown in Figure 8 (A), when the size of the synthesized second image 801a is changed to be larger than the state in Figure 7, the area indicated by the dotted line 802a in the first image is highlighted as the area of the corresponding depth distance.

In this way, by highlighting the area of the first image that corresponds to the depth distance, it is easy to visually recognize that the size of the changed second image is the size when it is placed at the same depth distance as the highlighted area of the first image. Also, as in Figure 8 (B), when the depth distance at which the second image is synthesized is brought closer, the size of the synthesized second image 801b changes, and the dotted line 802b indicating the subject position at the same distance as the depth distance in the first image is changed accordingly. Note that in Figure 8 (B), the depth distance at which the second image is synthesized is brought closer to the imaging device (camera) compared to Figure 8 (A), so it is displayed larger and synthesized.

As a method of displaying the depth distance to be synthesized, the numerical value of the depth distance to be synthesized can be displayed in one of the areas on the touch panel display, as shown in 803a or 803b in Figure 8.

FIG. 9 is a diagram for explaining another method of displaying the depth distance synthesized in the embodiment.
9, distance bars 901a and 901b are displayed, which indicate the maximum and minimum depth distances that can be arranged. By displaying the depth distance as a bar in this way, it is possible to show the user the depth distance for composition by displaying the position of the currently set depth distance for composition.
FIG. 9A shows an example in which the second image is arranged at the same distance as in FIG. 8A, and FIG. 9B shows an example in which the second image is arranged at the same distance as in FIG. 8B.

The first image may be converted into a luminance signal and a pseudo color corresponding to the depth distance may be superimposed on the image, and the distance bars 901a and 901b may be displayed as similar pseudo color bars to clearly show the distance.
Furthermore, after changing the depth distance for synthesis, it is necessary to provide an operation for returning the size of the synthesized second image to the size at the distance on the synthesis position coordinates on the first image.

For example, after changing the depth distance, the depth distance to be synthesized may be reset to the depth distance of the first image on the synthesis coordinates by detecting a double tap operation in the synthesized second image area. At the same time, the size of the second image may be changed to match the depth distance of the first image on the synthesis coordinates.

In step S210, the control unit 11 determines whether to move the coordinate positions in the X and Y directions on the first image relative to the second image. If it is determined that the movement should be performed, the movement is performed without changing the depth distance (depth position) to be combined, that is, without changing the size of the second image after the depth distance to be combined is changed. For example, if two points in the combined second image area are touched simultaneously, it is determined that this is an operation to move the second image, and the second image is moved by moving the two points simultaneously. In this case, the movement is performed while maintaining the relationship between the reference point and the two selected points. After the movement, the process returns to step S204, and the processes from step S204 onwards are repeated.

In step S211, the control unit 11 determines whether the user has touched the area 402 of the image displayed on the touch panel display to select a composite image, and thereby determines whether or not to additionally composite another image into the first image. If it is determined that additional composite is to be performed, the process returns to step S203, and the process from step S203 onward is executed. If additional composite is not to be performed, the process waits for input from the user until the generation of the composite image is completed.

When deleting the composite second image from the first image, two points in the second image composited with the first image are touched and dragged out of the first image area. Alternatively, when the second image composited with the first image is touched and then moved at a high speed equal to or faster than a predetermined speed, it may be determined as a deletion operation and deleted.
As described above, according to this embodiment, by using the shooting information and the like attached to the captured image, it becomes possible to more freely synthesize the size of a subject existing at a desired position in the image.

The present invention has been described in detail above based on its preferred embodiments, but the present invention is not limited to the above embodiments. Various modifications are possible based on the gist of the present invention, and these modifications are not excluded from the scope of the present invention.
A computer program for implementing all or part of the control in this embodiment may be supplied to the image processing device via a network or various storage media. Then, a computer (or a CPU, MPU, etc.) in the image processing device may read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention.

REFERENCE SIGNS LIST 100 Computer 10 Input section 11 Control section 12 Calculation section 13 Storage section 14 Output section

Claims (13)

1. A method for compositing a second image onto a first image, comprising the steps of:
a position detection step of detecting a designated position on the first image for synthesizing the second image;
a composite display step of changing a size of the subject of the second image based on subject distance information of the first image at the specified position and size information of the subject of the second image so that the size corresponds to the subject distance of the first image at the specified position, and displaying the composite image;
a depth distance change detection step of detecting a depth distance change operation for disposing the second image at a depth distance different from the subject distance at the specified position on the specified position;
a size changing step of changing a size of the second image in response to the depth distance change operation;
a depth distance information display step of displaying changed depth distance information in response to the depth distance change operation;
having
The image synthesis method according to claim 1, wherein in said depth distance information display step, a region of said first image corresponding to said depth distance is highlighted . 2. The image synthesis method according to claim 1 , wherein the depth distance information display step displays a numerical value of the depth distance. 2. The image synthesis method according to claim 1 , wherein the depth distance is displayed as a bar in the depth distance information display step. The image synthesis method according to claim 1, characterized in that, in the synthesis display step, the second image is displayed in an area of the first image into which the second image is synthesized, the area being at the depth distance from the specified position, and the first image is displayed in an area in front of the depth distance from the specified position. The image synthesis method according to claim 1, characterized in that the depth distance change detection process detects that the depth distance change operation has been performed based on the time or pressure with which the user touches the touch panel display, or by touching multiple points. The image synthesis method according to claim 1, characterized in that in the depth distance change detection process, the depth distance is changed according to the time, pressure, or pinch operation of the user touching the touch panel display. The image synthesis method according to claim 1, further comprising a reference selection step of selecting a reference position in the second image, and synthesizing and displaying the selected reference position by matching it with the specified position on the first image. 8. The image synthesis method according to claim 7 , wherein in the reference selection step, the reference position of the second image is displayed. The image synthesis method according to claim 1, characterized in that the second image is a specified subject image extracted from a specified image. The image synthesis method according to claim 1, characterized in that the position detection process and the synthesis display process are performed periodically while the user is moving the specified position. The image synthesis method according to claim 1, characterized in that the position detection step detects a position where the user drags and drops the second image on a touch panel display. A computer program for controlling each step of the image synthesis method according to any one of claims 1 to 11 by a computer. An image synthesis device for synthesizing a second image on a first image, comprising:
a position detection means for detecting a designated position on the first image for combining the second image;
a composite display means for changing a size of the subject of the second image based on subject distance information of the first image at the specified position and subject size information of the second image so that the size corresponds to the subject distance of the first image at the specified position, and displaying the composite image;
a depth distance change detection means for detecting a depth distance change operation for disposing the second image at a depth distance different from the subject distance at the specified position on the specified position;
a size changing means for changing a size of the second image in response to the depth distance changing operation;
a depth distance information display means for displaying changed depth distance information in response to the depth distance change operation ,
an area of the first image corresponding to the depth distance is highlighted by the depth distance information display means ;

Images