JP6468484B2 - Image processing apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program for processing an image captured by an imaging unit.

  Virtual fitting has been proposed (see Patent Document 1), in which an image of a clothing try-on image is superimposed on a video image of a person photographed by a camera, and is actually used in stores and the like.

  In virtual fitting, a video camera is placed beside the large-screen display panel, and the image of a person standing in front of the display panel is displayed on the display panel. Can be displayed.

  If the camera is arranged near the center of the display panel, the display on the display panel is hindered. Therefore, the camera is generally arranged on the side of the display panel. However, if the camera is arranged on the side of the display panel, a person is photographed from an oblique direction. If the camera is left as it is, the image of the person displayed on the display panel is distorted. More specifically, a larger picture is taken on the side closer to the camera, and a smaller picture is taken on the side far from the camera. In order to correct such distortion, it is effective to perform trapezoidal correction using lens distortion correction and projective transformation (see Patent Documents 2 and 3).

JP 2005-136538 A International Publication 2012/056982 International Publication Number WO2011 / 033393

  When trapezoidal correction is performed, distortion occurs in the edge portion of the image, so the portion where the distortion occurs is trimmed. At the time of drawing, processing for enlarging the image is performed in order to adjust the size of the image to the original size. However, when such a trimming process is performed, the image range becomes narrower than the original photographed image, and the image range included in the corrected image changes depending on the distance from the camera. For this reason, it is difficult to generate a wide range of corrected images with high resolution by trapezoidal correction.

  SUMMARY An advantage of some aspects of the invention is that it provides an image processing apparatus, an image processing method, and an image processing program capable of generating a high-resolution and wide-range distortion-free image by simple processing. It is to be.

In order to solve the above-described problem, in one aspect of the present invention, a plurality of imaging units that are arranged in different places and each shoot a common subject,
An image correction unit that generates a plurality of corrected images by correcting distortion of a captured image of a subject captured by each of the plurality of shooting units;
The plurality of corrected images are overlapped based on a predetermined reference point in each corrected image, and one partial image is selected for the partial image in the overlapping range, and the partial image in the non-overlapping range is left as it is. An image composition unit for arranging and generating a composite image;
An image processing apparatus is provided that includes a display control unit that causes the display unit to display the composite image as a captured image of a subject.

  The image correction unit corrects distortion with the same size of the subject in each captured image based on the distance from the plurality of photographing units to the subject and the direction from the plurality of photographing units to the subject. The plurality of corrected images may be generated.

  The plurality of photographing units may photograph the subject such that the reference point of the subject coincides with the center of the photographed image.

  The reference point may be a feature point of the subject.

  The plurality of photographing units may be arranged along at least two of a left end side, a right end side, an upper end side, and a lower end side of the display unit.

The subject may be a person,
The display control unit may cause the display unit to display an image obtained by superimposing another image selected by the person on the composite image of the person.

In another aspect of the present invention, generating a plurality of corrected images in which distortions of captured images captured by a plurality of capturing units that are arranged at different locations and each capture a common subject are corrected,
The plurality of corrected images are overlapped based on a predetermined reference point in each corrected image, and one partial image is selected for the partial image in the overlapping range, and the partial image in the non-overlapping range is left as it is. Arranging to generate a composite image; and
Displaying the composite image as a photographed image of a subject on a display unit.

In another aspect of the present invention, generating a plurality of corrected images in which distortions of captured images captured by a plurality of capturing units that are arranged at different locations and each capture a common subject are corrected,
The plurality of corrected images are overlapped based on a predetermined reference point in each corrected image, and one partial image is selected for the partial image in the overlapping range, and the partial image in the non-overlapping range is left as it is. Arranging to generate a composite image; and
There is provided a computer-readable image processing program for causing a computer to execute the step of displaying the composite image as a captured image of a subject on a display unit.

  According to the present invention, it is possible to generate an image with high resolution and a wide range of distortion by simple processing.

1 is a block diagram showing a schematic configuration of an image processing apparatus 1 according to an embodiment of the present invention. The perspective view of the image processing apparatus 1 of FIG. 3 is a flowchart showing an example of processing operation of the image processing apparatus 1. (A) And (b) is a figure which shows typically the image range before and behind performing a distortion correction process. (A)-(d) is a figure explaining the production | generation method of a synthesized image.

  Hereinafter, embodiments of the present invention will be described in detail. FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus 1 according to an embodiment of the present invention, and FIG. 2 is a perspective view of the image processing apparatus 1 of FIG.

  The image processing apparatus 1 according to the present embodiment includes a plurality of imaging units 2, an image correction unit 3, an image composition unit 4, a display control unit 5, and a display panel 6.

  For example, as shown in FIG. 2, the plurality of photographing units 2 are arranged at the left and right end portions and the upper and lower end portions of the display panel 6 (display portion). Note that the arrangement location of FIG. 2 is an example, and the arrangement location of the imaging unit 2 is not particularly limited. Further, as will be described later, the area of the finally obtained corrected image can be increased as the number of photographing units 2 is increased. Therefore, the number of imaging units 2 may be set according to the required area.

  The plurality of imaging units 2 have an imaging device such as a complementary metal oxide semiconductor (CMOS) sensor and a charged coupled device (CCD) and an optical system that performs focus adjustment, and can capture not only still images but also moving images. It is. As this camera, a camera as small as a camera mounted on a mobile phone or a smartphone is sufficient, and the aperture of each photographing unit 2 may be several mm or less.

  In FIG. 2, an example in which the four photographing units 2 are attached to the display panel 6 is shown, but a plurality of photographing units 2 may be integrated into the frame portion of the display panel 6. In this case, since it is not necessary to attach the plurality of photographing units 2 to the casing of the display panel 6, the appearance is neat and the design is improved.

  In the present embodiment, it is assumed that shooting is performed by the plurality of shooting units 2 in a state where a user as a subject is arranged at a predetermined position on the front side of the display panel 6. This is because if the user's location changes every time shooting is performed, the distortion correction processing cannot be performed with high accuracy. Therefore, for example, it is desirable to provide a mark or the like indicating the user's location on the floor surface on the front side of the display panel 6.

  The image correction unit 3 corrects distortion of the captured image of the subject captured by each of the plurality of imaging units 2. More specifically, trapezoidal correction is performed on a plurality of captured images captured by the plurality of capturing units 2 to generate a plurality of corrected images. More specifically, the image correction unit 3 has the same subject size in each captured image and distortion based on the distances from the plurality of photographing units 2 to the subject and the directions from the plurality of photographing units 2 to the subject. A plurality of corrected images are generated.

  The image synthesizing unit 4 superimposes a plurality of correction images based on a predetermined reference point in each correction image, selects one partial image among the partial images in the overlapping range, and a non-overlapping portion Images are arranged as they are to generate a composite image. The processing operations of the image correction unit 3 and the image composition unit 4 will be described in detail later.

  The display control unit 5 displays the composite image on the display panel 6 as a captured image of the subject. The display panel 6 is a liquid crystal display device, an organic EL (ElectroLuminescent) display, a plasma display, or the like.

  In the example of FIG. 2, an image processing box having the image correction unit 3, the image composition unit 4, and the display control unit 5 is arranged on the back side of the display panel 6 so that the user cannot visually recognize the image processing box. The image correction unit 3, the image composition unit 4, and the display control unit 5 may be provided separately from each other. Further, the image correction unit 3, the image composition unit 4, and the display control unit 5 may be incorporated in the display panel 6.

  FIG. 3 is a flowchart showing an example of the processing operation of the image processing apparatus 1. First, when the user is placed at a predetermined position on the front side of the display panel 6, the plurality of photographing units 2 starts photographing in accordance with a user instruction (step S1). Since the plurality of photographing units 2 photograph the user who is the subject from an oblique direction, the plurality of photographed images are all distorted. More specifically, the side closer to the photographing unit 2 is photographed larger, and the side far from the photographing unit 2 is photographed smaller.

  Each of the plurality of imaging units 2 captures the subject such that the reference point of the user as the subject matches the center of the captured image.

  Next, the image correcting unit 3 performs distortion correction on the plurality of captured images captured by the plurality of capturing units 2 to generate a plurality of corrected images (step S2). Here, for example, known trapezoidal correction is performed. As described above, when trapezoidal correction is performed, distortion occurs in the edge portion of the corrected image, and thus a process for trimming the distortion image in the edge portion is necessary. As a result, the image range of the corrected image becomes narrower than the image range of the photographed image before the distortion correction processing is performed.

  FIG. 4 is a diagram schematically showing an image range before and after performing the distortion correction processing. As shown in FIG. 4A, if the photographed image is a rectangular shape, a part of the captured image is trimmed to a trapezoidal shape after the distortion correction processing. Also, as shown in FIG. 4B, if the captured image is a trapezoidal shape, a part of the captured image is trimmed to a rectangular shape after the distortion correction processing. Thus, by performing distortion correction processing such as trapezoidal correction, the image range of the corrected image becomes narrower than the image range of the original captured image.

  Next, the image composition unit 4 generates a composite image using a plurality of corrected images corrected by the image correction unit 3 (step S3).

  FIG. 5 is a diagram for explaining a synthetic image generation method. In this example, as shown in FIG. 5A, an example in which a composite image is generated using four corrected images g1 is shown.

  The image composition unit 4 first selects one of a plurality of corrected images. All of the selected corrected images g1 become a part of the composite image (FIG. 5B). Next, another correction image g2 that has not yet been selected is selected, and its reference point is overlapped with the reference point of the correction image that has been selected previously (FIG. 5C). Here, the reference point is a feature point on the subject. When two corrected images are superimposed, an overlapping range and a non-overlapping range are generated. For the overlapping range, the previously selected correction image g1 is selected, and for the range that does not overlap with the previously selected correction image, the correction image g2 selected later is selected, and the composite image is compared with FIG. 5B. To spread.

  The above processing is repeated in order until all the corrected images g1 to g4 are selected (FIG. 5D). That is, the image composition unit 4 sequentially selects a plurality of correction images one by one, combines the correction image selected earlier with priority, and overlaps the correction image selected earlier in the correction image selected later. A composite image is generated by synthesizing the partial images in the range that has not been processed.

  As can be seen from FIG. 5, each time a corrected image is newly selected, the area of the composite image increases. Therefore, when it is desired to increase the area of the composite image, the number of photographing units 2 may be increased.

  The composite image generated in step S3 in FIG. 4 is drawn on the display panel 6 by the display control unit 5 (step S4). The composite image drawn on the display panel 6 is corrected for distortion and includes a wide area around the subject.

  The composite image of the subject generated according to the present embodiment can be used for, for example, virtual fitting. When performing virtual fitting, the user selects clothes to try on while the composite image is displayed on the display panel 6. The clothes may be selected by providing a touch panel function on the display panel 6 and the user touching and selecting desired clothes according to the display on the display panel 6. Alternatively, the user's voice may be recognized and clothes may be selected. Alternatively, clothing may be selected through a remote controller or another person.

  The display control unit 5 displays an image of clothes selected by the user by some means so as to overlap the above-described composite image. Since the composite image generated by the above-described processing procedure is corrected for distortion, it is possible to suppress positional deviation and size deviation from the clothing image, and to display a realistic image that the user is actually wearing clothing. Can be displayed.

  In addition, the composite image generated according to the present embodiment can be applied to purposes other than virtual fitting. The present invention can also be applied to the purpose of virtually changing accessories such as earrings and necklaces. Also, the subject is not necessarily only a human being. For example, the present invention can be applied to the purpose of virtually changing costumes for pets such as dogs and cats. Further, the present invention can be applied not only to animals such as people and pets but also to the purpose of virtually changing the interior of a room.

  As described above, in the present embodiment, a plurality of correction images obtained by shooting a common subject with a plurality of shooting units 2 arranged at different locations and correcting distortion of a plurality of shot images of each shooting unit 2 are used. Since one partial image is selected for the partial images in the overlapping range and the partial images in the non-overlapping range are arranged as they are to generate a combined image, a wide range combined image without distortion can be generated.

  Therefore, when the composite image is used for virtual fitting, when the image of the clothes is superimposed on the composite image of the user, the positional deviation and the size deviation can be suppressed, and it seems that the user actually wears the clothes. A realistic image can be displayed.

  At least a part of the image control apparatus 1 described in the above-described embodiment may be configured by hardware or software. When configured by software, a program for realizing at least a part of the functions of the image control apparatus 1 may be stored in a recording medium such as a flexible disk or a CD-ROM, and read and executed by a computer. The recording medium is not limited to a removable medium such as a magnetic disk or an optical disk, but may be a fixed recording medium such as a hard disk device or a memory.

  In addition, a program that realizes at least a part of the functions of the image control apparatus 1 may be distributed via a communication line (including wireless communication) such as the Internet. Further, the program may be distributed in a state where the program is encrypted, modulated or compressed, and stored in a recording medium via a wired line such as the Internet or a wireless line.

  The aspect of the present invention is not limited to the individual embodiments described above, and includes various modifications that can be conceived by those skilled in the art, and the effects of the present invention are not limited to the contents described above. That is, various additions, modifications, and partial deletions can be made without departing from the concept and spirit of the present invention derived from the contents defined in the claims and equivalents thereof.

  DESCRIPTION OF SYMBOLS 1 Image processing apparatus, 2 imaging | photography part, 3 image correction part, 4 image synthetic | combination part, 5 display control part, 6 display panel

Claims (8)

A plurality of photographing units arranged in different places on the frame portion of the display panel , each of which captures a common subject,
An image correction unit that generates a plurality of corrected images by correcting distortion of a captured image of a subject captured by each of the plurality of shooting units;
The plurality of corrected images are overlapped based on a predetermined reference point in each corrected image, and one partial image is selected for the partial image in the overlapping range, and the partial image in the non-overlapping range is left as it is. An image composition unit for arranging and generating a composite image;
An image processing apparatus comprising: a display control unit configured to display the composite image on a display unit as a captured image of a subject.   The image correction unit corrects distortion with the same size of the subject in each captured image based on the distance from the plurality of photographing units to the subject and the direction from the plurality of photographing units to the subject. The image processing apparatus according to claim 1, wherein the plurality of corrected images are generated.   The image processing apparatus according to claim 1, wherein the plurality of photographing units photograph the subject such that a reference point of the subject coincides with a center of the photographed image.   The image processing apparatus according to claim 3, wherein the reference point is a feature point of a subject.   5. The image processing apparatus according to claim 1, wherein the plurality of photographing units are arranged along at least two of a left end side, a right end side, an upper end side, and a lower end side of the display unit. The subject is a person,
The image processing apparatus according to claim 1, wherein the display control unit displays an image obtained by superimposing another image selected by the person on the composite image of the person on the display unit. Generating a plurality of corrected images obtained by correcting distortions of captured images taken by a plurality of photographing units that are arranged at different positions on the frame portion of the display panel and each photograph a common subject;
The plurality of corrected images are overlapped based on a predetermined reference point in each corrected image, and one partial image is selected for the partial image in the overlapping range, and the partial image in the non-overlapping range is left as it is. Arranging to generate a composite image; and
Displaying the composite image as a photographed image of a subject on a display unit. Generating a plurality of corrected images obtained by correcting distortions of captured images taken by a plurality of photographing units that are arranged at different positions on the frame portion of the display panel and each photograph a common subject;
The plurality of corrected images are overlapped based on a predetermined reference point in each corrected image, and one partial image is selected for the partial image in the overlapping range, and the partial image in the non-overlapping range is left as it is. Arranging to generate a composite image; and
A computer-readable image processing program for causing a computer to execute the step of displaying the composite image on a display unit as a photographed image of a subject.

Images