JP5593243B2 - Stereo image processing system, stereo image processing apparatus, and stereo image processing method - Google Patents

Description

  The present invention relates to a stereoscopic image processing technique, and more particularly to a stereoscopic image processing technique for effectively displaying a stereoscopic image on a monitor based on acquired data.

  In recent years, with the spread of stereoscopic image display devices, the development of stereoscopic image processing techniques for generating and processing stereoscopic images has become active.

  In the technique described in Patent Document 1, a template image storage step for storing left and right template images indicating a shadow pattern or contour pattern of a subject in a storage means by computer processing, and the original subject by two cameras. The left and right projection images obtained by photographing from two viewpoints or two directions are input to a computer, and the left and right template images are read from the storage means by computer processing, and the left template image is converted into the left projection image. Compositing and correcting or supplementing the left projection image, forming the left composite image when the subject is viewed from one viewpoint or direction, and combining the right template image with the right projection image, The right projection image is corrected or supplemented, and the right composite image when the subject is viewed from the other viewpoint or direction Formed, and forms a composite image of the left and right granted shading or contour. That is, the present invention discloses an image composition for composing and correcting and correcting another image on the left and right images.

  In general, a stereoscopic video having parallax information is generated from a live-action two-dimensional video that generates a stereoscopic video based on the depth information extracted from the two-dimensional video signal and the two-dimensional video signal. Can do. However, with the above-described conventional technology, it is impossible to view a video that has been viewed in a large amount.

  The technique described in Patent Literature 2 below is a method for generating a stereoscopic video based on a two-dimensional video and information for stereoscopic viewing thereof, and a virtual camera based on the viewer's head position or operation information input by the viewer. 3D based on the 2D image closest to the virtual camera position and the stereoscopic viewing information among a plurality of 2D images corresponding to the images determined by a plurality of cameras arranged at predetermined intervals. Generate video. More specifically, two-dimensional photographs are taken from a plurality of angles, thereby forming left and right composite images. Using a three-dimensional display, the left and right composite images are viewed at the same time with the left and right eyes. Alternatively, the supplemented subject can be shown three-dimensionally.

  With the above configuration, since a plurality of two-dimensional images (a plurality of cameras) corresponding to those photographed by a plurality of cameras arranged at predetermined intervals are used, it is possible to view a video that has been viewed in a large amount. . Since the stereoscopic video is generated based on the two-dimensional video closest to the virtual camera position among the plurality of two-dimensional video and the stereoscopic vision information, the video is smoothly switched.

JP 2008-210405 A JP 2004-200814 A

  However, the technique of the above-mentioned Patent Document 1 does not shoot from the front and back, and cannot create all 360-degree stereoscopic images. The image stored as the template is used for composition from above, and does not generate a 3D video based on the captured data.

  In the technique disclosed in Patent Document 2, a two-dimensional photograph is taken from a plurality of angles to make it appear three-dimensional, and a three-dimensional image is not created based on a two-dimensional image and a three-dimensional image.

  An object of the present invention is to generate a three-dimensional image with high accuracy and simple processing based on an acquired actual two-dimensional video and three-dimensional video.

  According to one aspect of the present invention, the first and second 3 including distance data between the first position and the second position that are different from the first position and acquired at the same timing. The first and second two-dimensional data not including distance data between the dimension data and the subject are input from the image input unit, and the input first and second three-dimensional data and the first and second There is provided a stereoscopic image processing apparatus having an image processing unit that generates stereoscopic image data by combining the second two-dimensional data.

  In addition, the present invention is the same by at least two or more photographing devices of the first photographing device provided at the first position and the second photographing device provided at a second position different from the first position. The first and second three-dimensional data including the distance data between the subject and the first and second two-dimensional (decorative image) data not including the distance data between the subjects, acquired at the timing. It has an image processing unit that generates stereo image data by synthesizing the input first and second three-dimensional data and the first and second two-dimensional data as an input from an image input unit. This is a featured stereoscopic image processing apparatus. A subject obtained at the same timing by at least two photographing devices of a first photographing device provided at a first position and a second photographing device provided at a second position different from the first position. The first and second three-dimensional data including the distance data between and the first and second two-dimensional (decorative image) data not including the distance data between the subject are input from the image input unit. By doing so, it is possible to reduce the burden of processing for generating stereoscopic image data.

  The present invention also provides first and second three-dimensional data including distance data between the first position and the second position that are different from the first position and acquired at the same timing. First and second two-dimensional (decorative image) data not including distance data between the subject is input from the image input unit, and the input first and second three-dimensional data and the second A stereoscopic image processing apparatus having an image processing unit for generating stereoscopic image data by combining the first and second two-dimensional data, and the first and second imaging apparatuses, Each of the two photographing apparatuses is a stereoscopic image processing system including a 2D camera and a 3D camera that acquire 2D data and 3D data, respectively.

  According to another aspect of the present invention, first and second three-dimensional data including distance data between a first position and a second position different from the first position at the same timing. And acquiring the first and second two-dimensional data not including the distance data between the subject and the input first and second three-dimensional data and the first and second There is provided a stereoscopic image processing method comprising: an image processing step of generating stereoscopic image data by combining the two-dimensional data.

  The present invention may be a program for causing a computer to execute the above method, or a computer-readable storage medium for recording the program.

  According to the stereoscopic image processing technique of the present invention, it is possible to reduce the burden of processing for creating a three-dimensional stereoscopic image. In addition, since a stereoscopic image is created from an actual 2D video and a 3D video, a beautiful stereoscopic image can be created.

It is a functional block diagram which shows the example of 1 structure of the stereo image processing system by one embodiment of this invention. It is an external view which shows the example of 1 structure of the imaging device in the stereo image processing system by this Embodiment. It is a functional block diagram which shows the example of 1 structure of the image processing apparatus in the stereo image processing system by this Embodiment. It is a figure which shows an example of matching of an example of 2D data and 3D data, and 2D data and 3D data in the stereoscopic image processing system by this Embodiment. It is a flowchart figure which shows the flow of the stereo image processing by this Embodiment. It is a flowchart figure which shows the flow of the synthetic | combination process of a several image among the stereo image processing by this Embodiment. It is a flowchart figure which shows the flow of the complementation process of an image among the stereo image processing by this Embodiment. It is a figure which shows the mode of the three-dimensional image process by this Embodiment. It is a figure which shows the mode of the image complementation process by this Embodiment.

  In this specification, the 3D camera is, for example, a device that measures the depth by irradiating a subject with infrared rays and acquires the reflection data thereof to acquire 3D data. For example, SwissRanger (Mesa Imaging Corp.) (Registered trademark) SR4000 can be used. For this 3D camera, you can refer to http://www.mesa-imaging.ch/prodview4k.php. This 3D camera is necessary for 3D image generation. It is a camera (photographing device) for acquiring data (3D data) of a distance from a subject.

  On the other hand, the 2D camera refers to a digital camera or the like for capturing a general plane (two-dimensional) image (2D image).

  However, the 3D data output from the 3D camera often includes error data due to the accuracy of the device and the like, and when used as it is, there is a problem that the 3D data does not become clean 3D data.

  In many cases, the resolution of the 3D camera does not match the resolution of the decoration 2D image data of the 2D camera, and when the 3D data and the 2D data are combined, an image of a portion that is not necessary for stereoscopic image display is displayed. There is also a problem that the image is displayed in three dimensions.

  Hereinafter, a stereoscopic image processing technique capable of 3D display with a high visual effect according to the present embodiment will be described.

  FIG. 1A is a functional block diagram showing a configuration example of a stereoscopic image processing system according to an embodiment of the present invention. FIG. 1B is a diagram showing an example of a positional relationship when the system of FIG. 1A is viewed from above.

  As shown in FIG. 1A, the stereoscopic image processing system A includes a first imaging device (1) 101, a second imaging device (2) 102, an image processing device 107, a display device 108, have. The display device 108 is an arbitrary device provided in the stereoscopic image processing system A.

  The first photographing apparatus (1) 101 includes a 2D camera (web camera or the like) 103 and a 3D camera 104, and the second photographing apparatus 102 similarly includes a 2D camera 105 and a 3D camera 106. ing. The 3D cameras 104 and 106 measure the distance from the 3D cameras 104 and 106 to the shooting target H (here, an example in which a person is a target) and acquire data for creating a three-dimensional shape of the shooting target. Equipment. The 2D cameras 103 and 105 are devices that take a two-dimensional picture of the subject H, and a general digital camera can be used as long as the device can take a digital image.

  The person H to be photographed stands so as to be sandwiched between the first photographing device (1) 101 and the second photographing device (2) 102, so that the first photographing device (1) 101 can be viewed from the back. The image and the second image taking device (2) 102 take images from the front and back (see FIG. 1B).

  In the present embodiment, the first image capturing device (1) 101 and the second image capturing device (2) 102 have an example in which a 2D camera and a 3D camera are separately provided. You may use the cameras D1 and D2 which integrated the 3D camera function. Further, a front image and a back image may be sequentially photographed while moving one photographing device 101 back and forth, and the number of photographing devices is increased to photograph a subject from a plurality of angles of 3 or more. Also good. You may make it image | photograph the image | video in a different position by rotating an imaging device.

  The first imaging device (1) 101 and the second imaging device (2) 102 are connected to the interface unit 113 of the image processing device 107 via interface units 111 and 112, respectively (wired or wireless connection). The image processing apparatus 107 performs correction, complementation, and composition processing on the image data sent from the first imaging apparatus (1) 101 and the second imaging apparatus (2) 102. As a result of the image processing by the image processing device 107, the surface image data (color, etc.) of the shooting target taken by the 2D cameras 103, 105 is combined with the three-dimensional shape of the shooting target shot by the 3D cameras 104, 106. Dimensional image data is completed.

  Based on the three-dimensional image data, a stereoscopic image is displayed on a display device 108 (for example, an LCD display) connected to the image processing device 107 via the interface units 114 and 115 (may be integrated with the image processing device 107). Is done. As another display means, 3D image data may be sent to a mobile terminal (not shown) connected to the network so that the 3D image can be confirmed on the display screen of the mobile terminal, and the display device is not limited.

  FIG. 2 is a diagram illustrating an external configuration example of the first imaging device (1) 101 and the second imaging device (2) 102. The 3D camera 104 or the 3D camera 106 is provided on the camera tripod 201, and the 2D camera 103 or the 2D camera 105 is mounted on the 3D camera 104 or the 2D camera 105. In short, a 2D camera and a 3D camera are provided at positions close to each other). As indicated by reference numerals D1 and D2, the 2D camera and the 3D camera may be integrated. In short, the 2D cameras 103 and 105 and the 3D cameras 104 and 106 may be the same camera or different cameras, but it is preferable that the positions of the two cameras are substantially the same.

  FIG. 3 is a functional block diagram showing a configuration example of a system centered on the image processing apparatus 107 according to the present embodiment. FIG. 4 is a diagram illustrating an example of captured image data and image information by the first imaging device (1) 101 and the second imaging device (2) 102. As illustrated in FIG. 3, the image processing apparatus 107 includes an image input unit 301, an image output unit 302, each image correction unit 303, a multiple image synthesis unit 304, an image complement processing unit 305, and a post-combination image correction. A processing unit 306 and a data storage unit 307 are included. The image processing unit 308 is configured by each of the image correction units 303, the multiple image synthesis unit 304, the image complement processing unit 305, and the post-combination image correction processing unit 306. Reference numerals 501 to 507 attached to the respective processing units 301 to 307 correspond to the respective processes in FIG.

  The image input unit 301 includes a front stereoscopic (data) 401 (FIG. 4) photographed by the 3D camera 106 (FIGS. 1 and 2) from the second photographing device (2) 102 and a front photograph (FIG. 4) photographed by the 2D camera 105 (FIG. 1). Data) 403 (FIG. 4) is received, and the back stereoscopic image (data) 402 (FIG. 4) and 2D camera 103 photographed by the 3D camera 104 from the first photographing device (1) 101 (data). 404 (FIG. 4) is received.

  Each image correction unit 303 removes noise and data error from the image data received from the image input unit 301, and the distance from the camera to the object to be photographed in the front solid (H 11) 401 and the back solid (H 13) 402 is reduced. A process of complementing the unacquired portion based on the surrounding distance data is performed.

  The multiple image composition unit 304 uses the image data processed by each image correction unit 303 to create a three-dimensional image data of the front image by pasting the front solid 401 and the front photograph 403, and the rear solid 402 and the rear photograph 404. Are combined to create three-dimensional image data of the rear image.

  FIG. 4A shows an example of 3D data, and FIG. 4B shows an example of 2D data. As shown in FIG. 4A, the front stereoscopic data 401 is image data obtained as a result of being photographed by the 3D camera 106 of the second photographing device (2) 102 placed in front of the subject H. Input from the image input unit 301. The front stereoscopic data 401 photographed by the 3D camera 106 has only a three-dimensional shape data, and does not have color elements such as clothes for the photographing target H. When the front image (H11) 407 of the imaging target H is rotated 90 degrees and viewed from the side surface (lateral), it has a form like a side image (H12) 408.

  As shown in FIG. 4B, front photograph data 403 photographed by the 2D camera 105 is also obtained from the second photographing device (2) 102. The front photograph data 403 is a two-dimensional photograph image having superficial colors such as a subject's face and clothes.

  Similarly, the back stereoscopic data (H13) 402 is obtained from the 3D camera 104 of the first photographing apparatus (1) 101 placed on the back of the subject H, and the back photograph data (H2) 404 is obtained from the 2D camera 103. Can be obtained.

  Table 1 is a table showing an example of a photo alignment parameter, and actually holds back data in addition to front data. Each of the front three-dimensional data 401 and the rear three-dimensional data 402 shown in FIG. 4 has parameters for aligning photographs as shown in Table 1.

For example, it has a parameter indicating the position of each coordinate of the vertices P ₁ , P ₂ , P ₃ , and P ₄ of the quadrangle shown in FIG. A front image H1 of a rectangular area defined by the vertices P ₁ , P ₂ , P ₃ , and P ₄ is extracted from the 2D image (front photo) data 403, and the front stereoscopic data 401 shown in FIG. Next, the subject H1 is aligned with the subject H11. This is applied to each of the rectangular areas defined by the rectangular areas P ₁ ′, P ₂ ′, P ₃ ′, and P ₄ ′ of the front stereoscopic data 401. Similarly, the rear stereoscopic data 402 is also aligned with the 2D image (rear photograph data) 404 so that the subject H2 matches H13. These alignment data are held in the data storage unit 307.

  The image complement processing unit 305 shoots the front image 3D image data 801 (FIG. 8) and the back image 3D image data 802 (FIG. 8) created by the multiple image composition unit 304 by stitching them on the left and right sides. Three-dimensional image data of the entire object is created, and a complementary process for smoothing the stitched portion is performed.

  The post-combination image correction processing unit 306 confirms whether or not there is a mismatch in the overall three-dimensional shape with respect to the three-dimensional image data of the entire photographing target created by the image complement processing unit 305, and the mismatch is found. If there is, format it.

  The three-dimensional image data created by the post-combination image correction processing unit 306 is output from the image output unit 302 to the display device 108. The display device 108 displays a three-dimensional image that can be viewed stereoscopically from all 360 degrees. Image data, image parameter information, parameter values used for various processes, and the like are stored in the data storage unit 307.

  FIG. 5 is a flowchart showing a flow of stereoscopic image processing according to the present embodiment. First, in step 501, an image photographed by the second photographing device (2) 102 is taken into the image input unit 301 as the front stereoscopic data 401 in FIG. The image photographed by the second photographing device (2) 102 at the same timing as the three-dimensional data is taken in (synchronized by a not-shown synchronizing means) is an image input unit 301 as a front photograph 403 in FIG. Is taken in. Furthermore, the image photographed by the first photographing apparatus (1) 101 is taken into the image input unit 301 as the back three-dimensional data 402 of FIG. An image photographed by the first photographing apparatus (1) 101 at the same timing as when the three-dimensional data is captured (synchronized by a not-shown synchronizing means) is an image input unit 301 as a rear photograph 404 in FIG. Is taken in.

  In step 502, each image correction processing in each image correction unit 303 in FIG. 3 is performed. If the front three-dimensional data 401 and the back three-dimensional data 402 acquired in step 501 include abnormal values or unexpected values, processing for removing those values is performed. When a part of the data is missing, a complement process is performed based on the surrounding data. For example, the distance to the subject is measured, the average value thereof is adopted, and data deviating from this average value is removed as the background. Further, with respect to the data of the subject after removing the background, the data protruding from the average value (uneven data) is removed as noise. As a result, the background is removed from the 3D data to extract the 3D data of the subject, and noise detected as fine irregularities in the extracted 3D data of the subject is also removed, and only the 3D data of the subject from which the noise has been removed is removed. Leave. That is, it is considered as noise based on comparison with a standard value such as comparison with a threshold value in 3D data, comparison with a median value, comparison with an average value, and comparison of difference between adjacent regions (periphery). Remove 3D data generated.

  A specific example of processing will be described. First, a threshold cut filter process based on Expression (1) is performed. By this processing, it is possible to cut a non-target part other than the subject such as a background part. Next, Median filter processing based on Expression (2) is performed. Even for a subject to be measured, distance data acquired based on, for example, infrared reflection by a 3D camera is not necessarily clean data, and a value that is partially larger than the distance as an intermediate value is often obtained. Therefore, a filtering process that does not use a portion that greatly deviates from the intermediate value distance (distance Z between the subject and the 3D camera) by the equation (2) is the subject portion. Although the reliability filter process based on the expression (3) is an arbitrary process, it is a process for removing invalid values due to camera-specific reliability. Next, as shown in Expression (4), smooth 3D data can be obtained by performing a process of averaging fine irregularities with nearby values using an average value filter. At this time, the entire surface is scanned, and if there is unevenness, a process for removing the unevenness is performed. Finally, 2D data is pasted onto 3D data to obtain, for example, colored 3D data. That is, based on the parameter 405 of Table 1 described above, as shown in FIG. 4, the front photographic data 403 is aligned and pasted to the front stereoscopic data 401.

  FIG. 8 is a diagram illustrating a transition state of a stereoscopic image formed along the stereoscopic image processing according to the present embodiment. The process up to the creation of the three-dimensional (3D) data 401 and 402 and the photograph (2D) data 403 and 404 in FIG. 8 is the same as the processing shown in FIG. By the process of aligning and pasting the front photograph data 403 on the front stereoscopic data 401, stereoscopic image composite data (front) 801 in which a color or the like is applied to the stereoscopic image is created. Similarly, stereoscopic image composite data (rear surface) 802 in which a color or the like is applied to a stereoscopic image is created by pasting the rear surface photograph data 404 on the rear surface stereoscopic data 402.

  Table 2 is a table holding a plurality of data alignment parameters 406a for the front, and has image information of a plurality of data alignment parameters. Actually, as shown in Table 3, in addition to the front data, the back data 406b is held.

  In step 503, the multi-image synthesis unit 304 performs multi-image synthesis processing. The parameters shown in Tables 2 and 3 are held in the stereoscopic image synthesis data (front) 801 and the stereoscopic image synthesis data (back) 802, respectively, and are used when a plurality of images are synthesized. The multiple image combining process is performed in accordance with the process shown in the flowchart of FIG. FIG. 6 is a diagram showing a detailed processing flow of the composite processing of a plurality of images in step 503 of FIG.

First, in step 601, based on the spatial coordinates X, spatial coordinates Y, and spatial coordinates Z of the parameters 406a and 406b in Tables 2 and 3, the positions of the stereoscopic image synthesis data (front) 801 and the stereoscopic image synthesis data (back) 802 are determined. Adjust each. Note that (X, Y) is a position coordinate in the two-dimensional plane, and Z is a position (distance) in a direction perpendicular to the two-dimensional plane. In step 602, the orientations of the stereoscopic image synthesis data (front) 801 and the stereoscopic image synthesis data (back) 802 are adjusted based on the rotation angles of the parameters 406a and 406b . In step 603, the data of the stereo image synthesis data (front) 801 and the stereo image synthesis data (back) 802 is analyzed to detect a synthesis location. In step 604, based on the combination location detected in step 603, the combination processing of the stereoscopic image combination data (front) 801 and the stereoscopic image combination data (back) 802 (in this case, combining by connecting the side surfaces). To create stereoscopic image synthesis data 803 associated with the side surface.

  Even after the processing in step 503, there may be a shortage of data at the synthesis location (side surface or the like). Therefore, in step 504, in this example, the stereoscopic image of the side surface portion is complemented in the image complementing processing unit 305. Details of the image complementing process are performed based on the process shown in the flowchart of FIG.

  As shown in FIG. 7, first, in step 701, a portion (in this case, a side surface portion) that needs to be supplemented with data when the stereoscopic image synthesis data (front) 801 and the stereoscopic image synthesis data (back) 802 are synthesized. To detect. If a complement location is detected, in step 702, the solid portion is first complemented. The three-dimensional interpolation processing is inferred from the three-dimensional shape of the three-dimensional image composition data (front) 801 and the three-dimensional image composition data (back) 802 (front three-dimensional data 401 and back three-dimensional data 402).

This complementing process will be described specifically by way of example.
1) Contour tracking processing is performed for points whose Z value is not INVALID (see, for example, http://homepage2.nifty.com/tsugu/sotuken/binedge/). By this processing, a plurality of contour positions are detected.
2) From the positions of a plurality of detected contours on the front surface and the back surface, the contours that are close to each other between the front surface and the back surface (front and back) are associated (see 803 in FIG. 8).
3) Corresponding points between the front surface and the back surface are calculated for all (X, Y) coordinates included in the contours associated in 2).
4) For the (X, Y) coordinates associated in 3), the corresponding 3D data points including the Z data are combined. (End of step 702)
5) Complement processing of 2D data (photograph part) to be pasted on the 3D data supplemented in the above 4) is performed (step 703). That is, when the three-dimensional part is complemented in step 702, there is no photograph data to be pasted at the complemented part, and therefore, the photograph part is supplemented in step 703. In the supplement processing of the photograph part, the color of the complement part is estimated from the front photograph data 403 and the back photograph data 404. For example, if the color of the front photo data 403 and the color of the back photo data 404 are the same, the same color is also applied to the complementary portion on the side. When the front and rear colors are different, the previous color may be gradually changed to the rear color.

  In step 704, a process of pasting the data of the photographic part supplemented in step 703 to the data of the three-dimensional part supplemented in step 702 is performed, and the three-dimensional image composition data 801 and the three-dimensional image composition data 802 are synthesized, thereby Completion processing of the copy ends.

  After step 504, in step 505, the post-combination image correction processing unit 306 performs correction processing on the stereoscopic image composite data 803. If there are abnormal values or unexpected values in the stereoscopic image composition data 803, those values are removed. In addition, when there is a large difference in color tone or shape between the stereoscopic image synthesis data (front) 801 and the stereoscopic image synthesis data (back) 802, and the stereo image synthesis data 803 deviates from the expected data, color correction or Perform shape correction.

Expression (5) is a specific example of an expression for performing the process of rounding the contour portion along the 4-point Bezier curve. FIG. 9 is a diagram showing an outline of processing for rounding a contour portion along a four-point Bezier curve. A process of rounding a quadrangular outline connecting four points in a square lattice composed of nine points P ₁₁ to P ₁₉ will be described. V1 is a vector of points to be joined (front), V4 is a vector of points to be joined (back), V2 and V3 are front and back correction vectors, and Vt is an output three-dimensional vector. When the inverse vector of the combined vector (vector average) of V ₁₁ , V ₁₂ , V ₁₃ , and V ₁₄ is obtained based on Expression (5), this is V ₂ or V ₃ . Based on the V ₂ or V _3, it is possible to round the contour of the side.

  Note that the complementing process in step 504 may be different depending on the type of subject. For example, in a rectangular parallelepiped such as a refrigerator, the side surface can be complemented with a flat surface, and in the case of a human being, the side surface is rounded to some extent. The third imaging device may perform processing related to the side surface, or the second imaging device of the two cameras may be installed at a position where the side surface can be imaged instead of the back surface so that the side surface can be measured. good.

  In step 506, the data of the point set subjected to the above processing is output to the data storage unit 307, and these data groups are stored in the storage unit 307.

  For example, in step 507, by reading the data of the stereoscopic image 803 from the data storage unit 307 and outputting it to the display device 108, the stereoscopic image can be displayed on the display device 108.

  Based on the data generated by the above processing, the stereoscopic image composition 803 can be displayed on the display device 108 as a final stereoscopic image 804 that can be stereoscopically viewed at 360 °.

  For example, by applying, for example, a church image as the background of the final stereoscopic image data 804 obtained by the image processing according to the present embodiment, an effect that an object (such as yourself) stands in the church is interactive. It becomes possible to provide attractive products (video, etc.).

  In the above-described embodiment, the configuration and the like illustrated in the accompanying drawings are not limited to these, and can be changed as appropriate within the scope of the effects of the present invention. In addition, various modifications can be made without departing from the scope of the object of the present invention.

  In addition, a program for realizing the functions described in the present embodiment is recorded on a computer-readable recording medium, and the program recorded on the recording medium is read into a computer system and executed to execute processing of each unit. May be performed.

  The present invention can be used as a three-dimensional stereoscopic image processing apparatus.

DESCRIPTION OF SYMBOLS 101 Image capturing apparatus 102 Image capturing apparatus 103 2D (Web) camera 104 3D camera 105 2D (Web) camera 106 3D camera 107 Image processing apparatus 108 Display apparatus 201 Tripod 301 Image input unit 302 Image output unit 303 Each image correction unit 304 Multiple image composition Unit 305 image complement processing unit 306 post-combination image correction processing unit 307 data storage unit 308 CPU
401 Front stereoscopic data 402 Rear stereoscopic data 403 Front photograph 404 Rear photograph 405 Photo alignment parameter
406a Positioning parameters for multiple data (front)
406b Multiple data alignment parameters (back)
501 Shooting process by photographing apparatus 502 Compositing process of each image 503 Compositing process of multiple images 504 Complementing process of image 505 Correcting process of post-compositing image 506 Saving process of result data 507 Display process of result data 601 Adjusting the position of each image Processing 602 Adjustment processing of the orientation of each image 603 Processing for detecting a combined portion of each image 604 Processing for combining each image 701 Processing for detecting a complementary portion of a composite image 702 Complement processing for a three-dimensional portion 703 Complement processing for a photographic portion 704 Complement portion Composition processing

Claims (12)

Same as at least two photographing devices of the first photographing device provided at the first position on the front side and the second photographing device provided at the second position on the back side different from the first position. The first and second three-dimensional data including the distance data between the subject and the first and second two-dimensional data not including the distance data between the subjects acquired at the timing from the image input unit. An image processing unit configured to generate stereoscopic image data by combining the input first and second three-dimensional data with the first and second two-dimensional data,
The image processing unit
The first three-dimensional data and the first two-dimensional data are aligned and pasted to create first stereoscopic image composite data, and the second three-dimensional data and the second 2D data are created. The second stereoscopic image composite data is created by aligning and pasting the dimension data,
A stereoscopic image processing apparatus that creates stereoscopic image composite data by adjusting the position and orientation of the first stereoscopic image composite data and the second stereoscopic image composite data and performing composite processing . The image processing unit
Each of the image correction units removes the error data related to the distance between the imaging device and the subject of the input first and second three-dimensional data by comparing with a standard value related to the distance. The stereoscopic image processing apparatus according to claim 1. The image processing unit
A two-dimensional plane in which the first and second three-dimensional data corrected by the respective image correction units and the first and second two-dimensional data are made to coincide with two-dimensional parameters for photographic alignment. 3. The stereoscopic image processing apparatus according to claim 2, wherein the first and second stereoscopic image composite data are created by combining the above. The image processing unit
Based on the respective spatial coordinates X, spatial coordinates Y, and spatial coordinates Z of the first and second three-dimensional data, the positions of the first and second stereoscopic image composite data are adjusted, respectively, Based on the respective rotation angles of the two-dimensional three-dimensional data, the direction of the first and second stereoscopic image synthesis data is adjusted, and the first and second stereoscopic image synthesis data are adjusted by adjusting the position and orientation. A part is detected, and the processing for synthesizing the first and second three-dimensional data at the detected synthesis part is performed to create stereoscopic image synthesis data associated with the synthesis part. Item 4. The stereoscopic image processing apparatus according to Item 3. A subject obtained at the same timing by at least two photographing devices of a first photographing device provided at a first position and a second photographing device provided at a second position different from the first position. The first and second three-dimensional data including the distance data between and the first and second two-dimensional data not including the distance data between the subjects are input from the image input unit and input. An image processing unit that generates stereoscopic image data by combining the first and second three-dimensional data and the first and second two-dimensional data;
The image processing unit
Each of the image correction units for removing the error data related to the distance between the photographing apparatus and the subject of the input first and second three-dimensional data by comparing with a standard value related to the distance;
A two-dimensional plane in which the first and second three-dimensional data corrected by the respective image correction units and the first and second two-dimensional data are made to coincide with two-dimensional parameters for photographic alignment. The first and second stereoscopic image composition data are created by combining the above,
Based on the respective spatial coordinates X, spatial coordinates Y, and spatial coordinates Z of the first and second three-dimensional data, the positions of the first and second stereoscopic image composite data are adjusted, respectively, Based on the respective rotation angles of the two-dimensional three-dimensional data, the direction of the first and second stereoscopic image synthesis data is adjusted, and the position of the first and second three-dimensional data is adjusted by adjusting the position and orientation. And performing the process of synthesizing the first and second three-dimensional data at the detected synthesis location, creating stereoscopic image synthesis data associated with the synthesis location,
In the synthesis process of the first and second three-dimensional data, the synthesis part is associated with the outlines at positions close to each other in the first and second three-dimensional data, and is included in the associated outlines Calculating corresponding points in the first and second three-dimensional data with respect to the spatial coordinates (X, Y), and combining the first and second three-dimensional data using the spatial coordinates Z; A featured stereoscopic image processing apparatus. The image processing unit
When synthesizing the first and second three-dimensional data, a portion that requires three-dimensional interpolation is detected and detected from the three-dimensional shape of the first three-dimensional data and the second three-dimensional data. The stereoscopic image processing apparatus according to claim 5, further comprising: a stereoscopic image complementing processing unit that performs a complementing process of a stereoscopic image that assumes a three-dimensional shape of a portion where the stereoscopic complementing is necessary and uses the supplemental portion. . The stereoscopic image complement processing unit
Further, a process of complementing the first and second two-dimensional data at the location where the supplement is necessary with respect to the complemented first and second three-dimensional data, and pasting the complement to the complement part. The stereoscopic image processing apparatus according to claim 6, wherein the stereoscopic image processing apparatus is performed. The image processing unit
8. The post-combination image correction processing unit that performs a process of rounding a contour portion of a portion that has been subjected to the complementing processing in the stereoscopic image synthesis data supplemented by the image complementing processing unit. 3. A stereoscopic image processing apparatus according to 1. First and second three-dimensional data including distance data from the subject acquired at the same timing at a first position on the front side and a second position on the back side different from the first position; The first and second two-dimensional data not including the distance data between the subject are input from the image input unit, and the input first and second three-dimensional data and the first and second An image processing unit that generates stereoscopic image data by combining two-dimensional data, and the image processing unit aligns and pastes the first three-dimensional data and the first two-dimensional data Thus, the first three-dimensional image composition data is created, and the second three-dimensional image composition data is created by aligning and pasting the second three-dimensional data and the second two-dimensional data. , The first stereoscopic image composition data and the first A three-dimensional image processing apparatus for creating a stereoscopic image synthesizing data by adjusting to synthesis processing the position and orientation of the three-dimensional image synthesis data,
First and second imaging devices,
The first and second imaging devices each include a two-dimensional camera and a three-dimensional camera that acquire two-dimensional data and three-dimensional data, respectively.   10. The stereoscopic image according to claim 9, further comprising synchronization means for acquiring each of the first or second three-dimensional data and the first or second two-dimensional data at the same timing. Processing system. In the first position on the front side and the second position on the back side, which is different from the first position, the first and second three-dimensional data including distance data between the subject and the same timing are acquired. Obtaining the first and second two-dimensional data not including the distance data between the subject,
An image processing step of generating stereoscopic image data by synthesizing the input first and second three-dimensional data and the first and second two-dimensional data;
The image processing step includes
The first three-dimensional data and the first two-dimensional data are aligned and pasted to create first stereoscopic image composite data, and the second three-dimensional data and the second 2D data are created. The second stereoscopic image composite data is created by aligning and pasting the dimension data,
A stereoscopic image processing method, wherein the stereoscopic image composite data is created by adjusting the position and orientation of the first stereoscopic image composite data and the second stereoscopic image composite data and performing composite processing .   After each image correction step of removing the error data relating to the distance between the input position of the first and second three-dimensional data input and the position of the subject by comparing with a standard value, the first The stereoscopic image processing method according to claim 11, wherein the first three-dimensional data and the first two-dimensional data are synthesized.

Images