JP6863007B2 - Image composition programs, methods and equipment - Google Patents

Description

The present invention relates to an image composition program, an image composition method, and an image composition apparatus.

A plurality of frame images that are continuous in a time series are acquired from a moving image or a plurality of still images arranged in a time series, and at least two or more frame images out of the plurality of frame images are joined to generate one still image. The first technique to do is proposed.

In addition, camera shake correction is performed by estimating the trajectory of the camera work intended by the user from the trajectory of the motion of the imaging result detected from the motion between the fields or frames of the shooting result, and correcting the motion between the fields or frames. A second technique has also been proposed.

In addition, a third technique has been proposed in which the contents of images shot by a video camera or the like are classified according to the camera work at the time of shooting. In the third technique, the approximate movement direction of the image pattern is detected from the moving image, and the camera work is classified into one of three types of still, pan, and zoom from the detected approximate movement direction.

Japanese Unexamined Patent Publication No. 2005-210428 Japanese Unexamined Patent Publication No. 2008-005109 Japanese Unexamined Patent Publication No. 10-2433340

For structures such as tunnels and bridges, for example, work is regularly inspected for damage such as cracks on the wall surface of the tunnel and the floor plate of the bridge. In this work, the presence or absence of damage is usually visually inspected, but by taking a picture of the object from a position farther than the visual inspection and making it possible to perform the inspection by referring to the image obtained by the shooting. , It is expected to realize labor saving of work.

However, assuming that the object is photographed with a resolution that allows the presence or absence of damage to be confirmed, the size of the object is very large compared to the size of the damage, so the object is inevitably partially divided. Each image will be taken separately. As an example, the size of an object can reach several to several tens [m] for a crack width of 0.1 [mm]. Then, in order to grasp where the damage found on the image is located in the entire object, a plurality of images taken for each part of the object are combined into one image in which the entire object is the subject. Is required.

Regarding the composition of a plurality of images, the first technique obtains a translation vector by pattern matching of images and tracking of feature points to calculate the relative position of the images. However, a plurality of images obtained by photographing an object for each part include an image in which a background area does not exist in the image and a part of the object occupies the entire surface, and such an image is a pattern. There are few image features used for matching and feature point tracking. In the first technique, the accuracy of pattern matching and feature point tracking is significantly reduced for an image having poor image features, so that it is not possible to calculate the relative position with other images, or the calculation accuracy of the relative position is significantly reduced. descend.

The second technique detects motion vectors in each part of the image by tracking feature points, detects a group of motion vectors belonging to the background, and averages the motion vectors belonging to the detected group, thereby causing camera shake and a camera. The camera motion vector due to the work is calculated. However, the second technique does not function effectively because the detection accuracy of the motion vector is remarkably lowered in the image having poor image features, and the group of the motion vector belonging to the background cannot be detected in the image in which the background region does not exist.

Further, in the third technique, the approximate moving direction of the luminance pattern is obtained based on the sign of the product of the spatial differential value and the time differential value of the image. However, in an image having poor image features, a clear peak does not appear in the spatial differential value of the image, and the peak of the time differential value is interrupted in the time corresponding to the image having poor image features. Therefore, the third technique does not function effectively for an image having poor image features.

In one aspect, it is an object of the present invention to obtain an image of an object from a plurality of images even when a plurality of images of the object taken for each portion include an image having poor image features.

In one embodiment, among a plurality of images in which the object is photographed for each part, the object to be reflected in the comparative image is based on the image features of the processed image and the comparative image in which the processed image and the imaging order are continuous. Specify the positional relationship of the part of the object that appears in the processed image with respect to the part. In addition, the camera work is estimated from the positional relationship of the specified processed image. Then, based on the estimated camera work, the position of the portion of the object to be reflected in the image whose positional relationship cannot be specified is determined, and a plurality of images are combined.

As one aspect, there is an effect that an image of an object can be obtained from a plurality of images even when a plurality of images of the object taken for each portion include an image having poor image features.

It is a functional block diagram of an image synthesizer. It is an image figure which shows an example of the photograph of an object by an operator. It is an image diagram which shows a plurality of images when the object is photographed for each part, and an example of the photographing range of each image. It is an image diagram which shows an example of the pattern of a camera work. It is a chart which shows an example of the camera work pattern information. It is a schematic block diagram of a computer functioning as an image synthesizer. It is a flowchart which shows an example of image composition processing. It is an image diagram which shows the shooting range of each image and the shooting order of each image in chronological order when the object is photographed for each part. It is a flowchart which shows an example of image relative position estimation processing. It is a flowchart which shows an example of image relative position estimation processing 2. It is an image diagram which shows an example of the search range in image matching. It is an image diagram which shows an example of the calculation of relative coordinates and approximate direction in image matching. It is an image diagram explaining the process of matching the image of the shooting order = 2 with the image of the shooting order = 1. It is an image diagram explaining the process of matching the image of the shooting order = 3 with the image of the shooting order = 2. FIG. 5 is an image diagram illustrating a process of matching an image having a shooting order = 4 with an image having a shooting order = 3. FIG. 5 is an image diagram illustrating a process of matching an image having a shooting order = 5 with an image having a shooting order = 4. It is an image diagram explaining the process of matching the image of the shooting order = 5 with the image of the shooting order = 1. It is an image diagram explaining the process of matching the image of the shooting order = 6 with the image of the shooting order = 5. It is a flowchart which shows an example of the composite image generation processing. It is an image diagram explaining the process of creating the whole image by arranging only the image whose positional relationship with other image is specified. It is an image diagram explaining the process of determining the position of the image of the shooting order = 6. It is an image diagram which shows an example of the composite image finally obtained. It is an image diagram which shows an example of the 1st output image. It is an image diagram which shows an example of the 2nd output image. It is an image diagram which shows a plurality of images when the object is photographed for each part, and an example of the photographing range of each image.

Hereinafter, an example of the embodiment of the present invention will be described in detail with reference to the drawings. The image synthesizer 10 shown in FIG. 1 includes an image acquisition unit 12, a crack detection unit 14, a positional relationship identification trial unit 16, a camera work estimation unit 18, a storage unit 20 that stores camera work pattern information 22, and a positional relationship determination unit 24. , The image synthesizing unit 26 and the image output unit 28 are included.

In the present embodiment, as shown in FIG. 2, the worker 32 operates the photographing device 34 with respect to the inspection object 30 (hereinafter, simply referred to as “object 30”) such as the wall surface of the tunnel and the floor plate of the bridge. A plurality of images are taken for each part of the object 30. In FIG. 2, the cracks generated in the object 30 are indicated by the reference numerals “30A”. As shown by the broken line in FIG. 3, the operator 32 performs each shooting operation of the object 30 so that a part of the adjacent shooting ranges overlaps. When the shooting range shown by the broken line in FIG. 3 is shot in each shooting of the object 30, the images I1 to I12 shown in FIG. 3 are obtained.

Further, in the images I1 to I12 shown in FIG. 3, the operator 32 takes a picture of the object 30 a plurality of times while performing camera work in which the shooting range by the photographing device 34 is moved along the pattern 1 shown in FIG. Obtained at. However, the camera work when the worker 32 shoots the object 30 is not limited to the pattern 1, and the camera work pattern may be, for example, any of the patterns 1 to 20 shown in FIG. The camera work pattern is not limited to the patterns 1 to 20 shown in FIG.

As shown in FIG. 5 as an example, the camera work pattern information 22 stored in the storage unit 20 is a combination of moving directions of a shooting range in which individual patterns of camera work are switched in chronological order (the first direction in FIG. 5). It is represented by (D1) to the fourth direction (D4)). Further, in the camera work pattern information 22, different pattern numbers are assigned to the information representing individual patterns of the camera work.

The image acquisition unit 12 of the image compositing device 10 acquires a plurality of images (for example, images I1 to I12 shown in FIG. 3) for each part of the photographed object 30 from the photographing device 34. The crack detection unit 14 searches a plurality of images acquired by the image acquisition unit 12 for the presence or absence of image features (hereinafter, simply referred to as “cracks”) corresponding to damage such as cracks. Then, when the crack is detected by the search, the crack detection unit 14 detects the properties such as the width, length, and shape of the detected crack, and identifies the information indicating the property of the crack and the image in which the crack is detected. Output with the information to be.

The positional relationship specifying trial unit 16 sequentially extracts two images having a continuous shooting order from the plurality of images acquired by the image acquisition unit 12, and based on the image features of each of the two images taken out, the two images are described. An attempt is made to specify the positional relationship of the portion of the object 30 reflected in the image. The camera work estimation unit 18 is a worker 32 based on the positional relationship of a plurality of images whose positional relationship is specified by the positional relationship identification trial unit 16 and the camera work pattern information 22 stored in the storage unit 20. Estimates the camera work when the object 30 is photographed. The positional relationship determination unit 24 determines the position of an image whose positional relationship with another image has not been specified by the positional relationship identification trial unit 16 based on the camera work estimated by the camera work estimation unit 18.

The image synthesizing unit 26 has a plurality of images acquired by the image acquisition unit 12 based on the positional relationship of the plurality of images specified by the positional relationship identification trial unit 16 and the positions of the images determined by the positional relationship determination unit 24. Is combined into one image in which the entire object 30 is the subject. Further, the image synthesizing unit 26 converts the crack information output from the crack detecting unit 14 into a synthesized image (first output image) and an image before synthesizing in which cracks are detected (second output image), respectively. Superimpose.

The image output unit 28 outputs the first output image and the second output image generated by the image composition unit 26. The first output image and the second output image output by the image output unit 28 are visually confirmed by, for example, an operator 32 or the like, and are used for inspection of the object 30.

In FIG. 1, the positional relationship specifying trial unit 16 is an example of a specific unit in the present invention, the camera work estimation unit 18 is an example of an estimation unit in the present invention, and the positional relationship determining unit 24 and the image synthesizing unit 26 are This is an example of a determination unit in the present invention.

Further, in the present embodiment, the image synthesizer 10 is realized by, for example, the computer 50 shown in FIG. The computer 50 includes a CPU 52, a memory 54, a non-volatile storage unit 56, an input unit 58, a display unit 60, a read / write device (R / W) 62 that reads and writes data to and from a recording medium 64, and a communication I. / Includes F section 66. The CPU 52, the memory 54, the storage unit 56, the input unit 58, the display unit 60, the R / W 62, and the communication I / F unit 66 are connected to each other via the bus 68. The image compositing device 10 is capable of communicating with the photographing device 34 via the network connected to the communication I / F unit 66.

The storage unit 56 is realized by an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, or the like. The storage unit 56 stores an image compositing program 70 for causing the computer 50 to function as the image compositing device 10. The CPU 52 reads the image composition program 70 from the storage unit 56, expands the image composition program 70 into the memory 54, and sequentially executes each process included in the image composition program 70.

The image composition program 70 includes an image acquisition process 72, a crack detection process 74, a positional relationship identification trial process 76, a camera work estimation process 78, a positional relationship determination process 80, an image composition process 82, and an image output process 84. The CPU 52 operates as the image acquisition unit 12 shown in FIG. 1 by executing the image acquisition process 72. Further, the CPU 52 operates as the crack detection unit 14 shown in FIG. 1 by executing the crack detection process 74. Further, the CPU 52 operates as the positional relationship specifying trial unit 16 shown in FIG. 1 by executing the positional relationship specifying trial process 76. Further, the CPU 52 operates as the camera work estimation unit 18 shown in FIG. 1 by executing the camera work estimation process 78. Further, the CPU 52 operates as the positional relationship determining unit 24 shown in FIG. 1 by executing the positional relationship determining process 80. Further, the CPU 52 operates as the image synthesizing unit 26 shown in FIG. 1 by executing the image synthesizing process 82. Further, the CPU 52 operates as the image output unit 28 shown in FIG. 1 by executing the image output process 84.

As a result, the computer 50 that executes the image composition program 70 functions as the image composition device 10. The image composition program 70 is an example of an image composition program according to the disclosed technology. Further, the storage unit 56 is provided with a camera work pattern information storage area 86, and the camera work pattern information storage area 86 stores the camera work pattern information 22. As a result, the storage unit 56 functions as the storage unit 20 shown in FIG.

The image synthesizer 10 can also be realized by, for example, a semiconductor integrated circuit, more specifically, an ASIC (Application Specific Integrated Circuit) or the like.

Next, as an operation of the present embodiment, with reference to FIG. 7, after the worker 32 operates the photographing device 34 to complete the work of photographing a plurality of images of the object 30 for each part of the object 30. The image composition process executed by the image composition apparatus 10 will be described.

In step 100 of the image composition process, the image acquisition unit 12 acquires a plurality of images (for example, images I1 to I12 shown in FIG. 3) obtained by photographing the object 30 for each part from the photographing apparatus 34. In step 102, the image acquisition unit 12 recognizes the shooting times of the plurality of images acquired in step 100, sorts the plurality of images in ascending order of shooting time, and assigns the image numbers in ascending order of shooting time to each image. Give. The shooting time of the image is set to EXIF (Exchangeable Image File Format) information if the image is in JPEG (Joint Photographic Experts Group) format, for example. By sorting in step 102, as shown in FIG. 8, the images I1 to I12 obtained by photographing the object 30 for each part are arranged in the order of the photographing time.

In step 104, the crack detection unit 14 searches for the presence or absence of cracks in each of the plurality of images acquired by the image acquisition unit 12. Then, when the crack is detected by the search, the crack detection unit 14 detects the properties such as the width, length, and shape of the detected crack, and identifies the information indicating the property of the crack and the image in which the crack is detected. The information is stored in the memory 54 or the like.

In the next step 106, the positional relationship identification trial unit 16 and the camera work estimation unit 18 perform image relative position estimation processing. Hereinafter, the details of the image relative position estimation process will be described with reference to FIG.

In step 120 of the image relative position estimation process, the positional relationship identification trial unit 16 sets 2 in the image number cn indicating the processed image (image to be processed) and points to the comparison image (image to be matched with the processed image). Set 1 for the image number rn. Further, the positional relationship specifying trial unit 16 identifies the variable i for identifying the order of the approximate movement direction (hereinafter referred to as “approximately direction”) of the shooting position, and each line along the main shooting direction of the camera work. 1 is set in each of the variables j of, and 0 is set in the number of images L (j) in the jth line.

In the present embodiment, the main shooting direction of the camera work corresponds to the first direction (D1) in the camera work pattern information 22 shown in FIG. 5, and the second direction (D2) in the camera work pattern information 22 is the camera work. It is called the sub-shooting direction of.

In the next step 122, the positional relationship identification trial unit 16 matches the processed image of the image number cn (hereinafter referred to as “processed image cn”) with the comparison image of the image number rn (hereinafter referred to as “comparative image rn”). I do. In this matching, for example, the processed image cn is moved in the X direction and the Y direction in 1-pixel increments within the search range 90 including all the relative positions P1 to P4 (see FIG. 11) of the processed image cn with respect to the comparative image rn. , The correlation evaluation value between the comparison image rn and the processed image cn is calculated at each relative position.

The relative position P1 is the position where the lower right corner of the processed image cn overlaps the upper left corner of the comparison image rn by a predetermined width (for example, one pixel width), and the relative position P2 is the position where the lower left corner of the processed image cn is the comparison image. It is a position that overlaps the upper right corner of rn by a predetermined width. In the relative position P3, the upper right corner of the processed image cn overlaps the lower left corner of the comparison image rn by a predetermined width, and in the relative position P4, the upper left corner of the processed image cn is only the lower right corner of the comparison image rn and the predetermined width. It is an overlapping position. Further, as the correlation evaluation value, any of various known parameters for evaluating the correlation between images can be used.

Further, the positional relationship specifying trial unit 16 extracts the maximum value from the correlation evaluation value calculated for each relative position in the search range 90, and determines whether or not the extracted maximum value is equal to or greater than a predetermined threshold value. When the maximum value of the correlation evaluation value is less than the threshold value, it can be judged that the processed image cn is not continuous with the comparison image rn, while when the maximum value of the correlation evaluation value is greater than or equal to the threshold value, the correlation evaluation value indicates the maximum value. It can be judged that the processed image cn is connected (continuous) to the comparison image rn at the relative position.

Therefore, when the maximum value of the correlation evaluation value is equal to or greater than the threshold value, the positional relationship specifying trial unit 16 uses the comparative image rn as the origin from the relative position where the correlation evaluation value shows the maximum value, as shown in FIG. Obtain the relative coordinates (Xcn, Ycn) of the processed image cn. Further, the positional relationship specifying trial unit 16 obtains the angle of the vector 92 from the origin toward the relative coordinates (Xcn, Ycn), and sets the approximate direction of the processed image cn with respect to the comparison image rn to the right and above based on the angle of the vector 92. , Left, or bottom. As an example, if the angle of the vector 92 is within the range of 315 ° to 45 °, the approximate direction is to the right, and if the angle is within the range of 45 ° to 135 °, the approximate direction is upward, and within the range of 135 ° to 225 °. If so, the approximate direction can be classified to the left, and if it is within the range of 225 ° to 315 °, the approximate direction can be classified to the upper.

When the matching between the processed image cn and the comparison image rn in step 122 is completed, in the next step 124, the positional relationship specifying trial unit 16 determines whether or not the processed image cn is continuous with the comparison image rn. If the processed image cn and the comparative image rn are images including the edges of the object 30 as in the images I1 and I2 shown in FIG. 8, the correlation evaluation value is based on the image features corresponding to the edges of the object 30. The maximum value of is greater than or equal to the threshold value.

However, for example, when the shooting range of the comparative image rn is located on the left side of the shooting range of the image I1 shown in FIG. 8, the processed image does not have the image feature corresponding to the edge of the object 30 in the comparative image rn. The maximum value of the correlation evaluation value between cn and the comparison image rn is less than the threshold value. In this case, the determination in step 124 is denied and the process proceeds to step 126. In step 126, the positional relationship specifying trial unit 16 registers the image number cn of the processed image in the position unknown table for registering the information of the image whose positional relationship with the other image is unknown.

In the next step 128, the positional relationship specifying trial unit 16 determines whether or not the image number cn has reached the total number of images acquired by the image acquisition unit 12. If the determination in step 128 is denied, the process proceeds to step 130, and in step 130, the positional relationship specifying trial unit 16 increments the image numbers rn and cn by 1 and returns to step 122. As a result, steps 122 to 130 are repeated while the determinations in steps 124 and 128 are denied, and the maximum value of the correlation evaluation value is equal to or greater than the threshold value while switching the processed image cn and the comparison image rn in ascending order of shooting order. The set of the processed image cn and the comparative image rn is searched.

The determination in step 128 is affirmed because there is a set of processed image cn and comparative image rn in which the maximum value of the correlation evaluation value is equal to or greater than the threshold value among the plurality of images acquired by the image acquisition unit 12. If you did not. In this case, it is possible that the shooting of the object 30 has failed, so processing to notify the error or the like is performed to end the image relative position estimation processing.

On the other hand, if the processed image cn and the comparative image rn are images including the edges of the object 30 as in the images I1 and I2 shown in FIG. 8, they are correlated from the image features corresponding to the edges of the object 30. The maximum value of the evaluation value exceeds the threshold value. As a result, the determination in step 124 is affirmed, and the process proceeds to step 132.

In step 132, the positional relationship specifying trial unit 16 sets the approximate direction of the processed image cn with respect to the comparative image rn classified in step 122 to the approximate direction Di. As an example, when the image I1 shown in FIG. 8 is the comparative image rn and the image I2 is the processed image cn, the approximate direction of the image I2 with respect to the image I1 is classified as “right” as shown in FIG. "Right" is set for D1. The positional relationship specifying trial unit 16 stores the set approximate direction Di in the memory 54 or the like together with the relative coordinates (X2, Y2) of the image I2 with the image I1 as the origin. Further, the positional relationship specifying trial unit 16 sets the image number rn of the comparison image corresponding to the origin of the relative coordinates of the processed image cn in the reference image REFcn, increments the number of images L (j) by 1, and increments the reference image REFcn. And the number of images L (j) is also stored in the memory 54 or the like. In the example shown in FIG. 13, the images I1 and I2 are images whose positional relationship is specified (the reference image REFcn and the relative coordinates (Xcn, Ycn) are stored in the memory 54 or the like), and the number of images L after incrementing. (1) = 1.

In the next step 134, the camera work estimation unit 18 extracts all the patterns whose first direction matches the approximate direction Di from the individual patterns of the camera work defined in the camera work pattern information 22. For example, in the example shown in FIG. 13, since the approximate direction D1 is “right”, a pattern in which the first direction is “right”, that is, pattern numbers 1, 2, 5, 6, 7, and 10 are extracted. Then, the camera work estimation unit 18 sets the pattern numbers of all the extracted patterns in the pattern table.

In step 136, the positional relationship identification trial unit 16 increments the image numbers rn and cn by 1 each. Further, in step 138, the positional relationship specifying trial unit 16 matches the processed image cn and the comparison image rn in the same manner as in step 122 described above. Then, in step 140, the positional relationship specifying trial unit 16 determines whether or not the processed image cn is continuous with the comparison image rn, as in step 124 described above.

If the processed image cn and the comparative image rn are images including the edges of the object 30 as in the images I2 and I3 shown in FIG. 8, the correlation evaluation value is based on the image features corresponding to the edges of the object 30. The maximum value of is greater than or equal to the threshold value. In this case, the determination in step 140 is affirmed, and the process proceeds to step 142. In step 142, the positional relationship specifying trial unit 16 determines whether or not the approximate direction of the processed image cn with respect to the comparative image rn matches the approximate direction Di.

As an example, when the image I2 shown in FIG. 8 is the comparative image rn and the image I3 is the processed image cn, the approximate direction of the image I3 with respect to the image I2 is classified as “right” as shown in FIG. It matches D1. In such a case, the determination in step 142 is affirmed and the process proceeds to step 144. In step 144, the positional relationship specifying trial unit 16 stores the relative coordinates (X3, Y3) of the image I3 with the image I2 as the origin in the memory 54 or the like. Further, the positional relationship specifying trial unit 16 sets the image number = 2 of the image I2 corresponding to the origin of the relative coordinates of the image I3 as the reference image REF3, increments the number of images L (1) by 1, and the reference image REF3. And the number of images L (1) is also stored in the memory 54 or the like. In the example shown in FIG. 14, the image I3 is added as an image in which the positional relationship with another image is specified (the reference image REFcn and the relative coordinates (Xcn, Ycn) are stored in the memory 54 or the like), and after incrementing. The number of images L (1) = 2.

When the process of step 144 is performed, the process proceeds to step 154. In step 154, the positional relationship specifying trial unit 16 determines whether or not the image number cn has reached the total number of images acquired by the image acquisition unit 12. If the determination in step 154 is denied, the process returns to step 136. As a result, while the determinations in steps 140 and 142 are affirmed and the determinations in steps 154 are denied, step 144 is repeated while incrementing the image numbers rn and cn by 1 respectively.

For example, when the image I3 shown in FIG. 8 becomes the comparative image rn and the image I4 becomes the processed image cn as the image numbers rn and cn are incremented, the outline of the image I4 with respect to the image I3 is as shown in FIG. The direction is classified as "right". In this example, the image I4 is added as an image whose positional relationship with another image is specified, and the number of images L (1) = 3 after incrementing.

By the way, when the worker 32 takes a picture of the object 30, for example, when the camera work corresponding to the pattern 2 shown in FIG. 4 is performed, the images I1 to I4 shown in FIG. The image I8 shown is taken. As a result, a scene occurs in which the image I4 becomes the comparative image rn and the image I8 becomes the processed image cn. In this case, although not shown, the approximate direction of the processed image cn with respect to the comparative image rn is classified as “down” and does not match the “right” of the approximate direction D1, so the determination in step 142 is denied and the process proceeds to step 146. Transition.

In step 146, the positional relationship specifying trial unit 16 increments the variables i and j by 1, sets the number of images L (j) to 1, and sets the approximate direction of the processed image cn with respect to the comparative image rn to the approximate direction Di. Set. Then, the positional relationship specifying trial unit 16 sets the set approximate direction Di, the relative coordinates (Xcn, Ycn) of the processed image cn with the comparison image rn as the origin, the reference image REFcn in which the image number rn of the comparison image is set, and The number of images L (j) is stored in the memory 54 or the like. For example, when the image I4 becomes the comparison image rn and the image I8 becomes the processed image cn, the approximate direction D2 is set to “down” and the number of images L (2) = 1.

In the next step 150, the camera work estimation unit 18 determines whether or not the number of pattern numbers registered in the pattern table is one. If the determination in step 150 is denied, the process proceeds to step 152.

In step 152, the camera work estimation unit 18 determines that the first to i directions defined in the camera work pattern information 22 are the approximate directions D1 to Di for each pattern whose pattern number is registered in the pattern table. It is determined whether or not all of them match. Then, the camera work estimation unit 18 deletes all the pattern numbers from the pattern table for the patterns in which the first direction to the i direction do not match the approximate directions D1 to Di.

For example, when the approximate direction D1 is "right" and the approximate direction D2 is "down", when the "right" of the approximate direction D1 is found, the pattern 1,2,5,6 in which the first direction is "right" is found. , 7 and 10 are registered in the pattern table. After that, when the "downward" of the approximate direction D2 is found, the patterns 6, 7, and 10 are deleted from the pattern table because the second direction is not "downward" and does not match the approximate direction D2.

When the process of step 152 is performed, the process proceeds to step 154. If the determination in step 150 is affirmed, step 152 is skipped and the process proceeds to step 154. Note that step 152 is not executed because the determination in step 150 is affirmed when the number of pattern numbers registered in the pattern table decreases to 1. When the number of pattern numbers registered in the pattern table becomes 1 in this way, the camera work estimation unit 18 completes the estimation of the camera work pattern.

On the other hand, when the worker 32 takes a picture of the object 30, for example, when the camera work corresponding to the pattern 1 shown in FIG. 4 is performed, the images I1 to I4 shown in FIG. 8 are taken in order, and then the image I5 is displayed. By being photographed, a scene occurs in which the image I4 becomes the comparison image rn and the image I5 becomes the processed image cn. In this case, as shown in FIG. 16 as an example, when the maximum value of the correlation evaluation value between the comparative image rn and the processed image cn becomes less than the threshold value, the determination in step 140 is denied and the process proceeds to step 148. In step 148, the positional relationship specifying trial unit 16 performs the image relative position estimation process 2. Hereinafter, the image relative position estimation process 2 will be described with reference to FIG.

In step 160 of the image relative position estimation process 2, the positional relationship identification trial unit 16 determines whether or not the image cn-1 is registered in the position unknown table. For example, when I5 shown in FIG. 8 becomes a processed image, the image cn-1 becomes I4. I4 is not registered in the unknown position table because its relative position with respect to I3 is determined as shown in FIG. Therefore, the determination in step 160 is denied, and the process proceeds to step 162. On the other hand, when the processed image is I7, the image cn-1 becomes I6. As shown in FIG. 18, I6 is registered in the unknown position table because the relative position with respect to I5 is not determined. Therefore, the determination in step 160 is affirmed, and the process proceeds to step 178.

In step 162, the positional relationship specifying trial unit 16 determines whether the number of images L (j) included in the approximate direction of the camera work being processed is 1. When L (j) is not 1, the switching portion of the camera work in which the main shooting direction is constant to the sub shooting direction, as in patterns 1, 3, 6, 8, 11, 13, 16 and 18 in FIG. As a result, the process proceeds to step 164. When L (j) is 1, the direction of the camera work in which the main shooting direction changes as shown in patterns 2, 4, 5, 7, 9, 10, 12, 14, 15, 17, 19, and 20 in FIG. As a changed part, the process proceeds to step 170.

In step 164, the positional relationship specifying trial unit 16 performs the same matching as in step 122 described above in order to obtain the relative position at the switching portion of the camera work in which the main shooting direction is constant to the sub shooting direction. And the comparison image cn-L (j) -1. For example, when the processed image is I5 shown in FIG. 8, cn = 5, j = 1, L (1) = 3, and the comparison image is I1, so I5 and I1 are matched.

In step 166, the positional relationship specifying trial unit 16 determines whether or not the processed image cn and the comparison image cn-L (j) -1 are continuous, as in step 124 described above. If the determination in step 166 is affirmed, the process proceeds to step 168.

In step 168, the positional relationship specifying trial unit 16 increments i and j by 1 and sets the number of images L (j) to 1. Further, the approximate direction Di is added with "oblique" in the approximate direction with respect to the processed image cn and the comparative image cn-L (j) -1. For example, when the image I5 shown in FIG. 8 is a processed image and the image I1 is a comparative image, as shown in FIG. 17, the approximate direction of the image I5 with respect to the image I1 is classified as "downward", and the approximate direction D2 , "Slanting" with "Slanting" added is set. The positional relationship specifying trial unit 16 stores the set approximate direction Di in the memory 54 or the like together with the relative coordinates (Xcn, Ycn) of the processed image with the comparison image as the origin. Further, the positional relationship specifying trial unit 16 sets the image number cn-L (j) -1 of the comparison image corresponding to the origin of the relative coordinates of the processed image cn as the reference image REFcn, and sets the reference image REFcn and the number of images L ( j) is also stored in the memory 54 or the like.

In step 170, the camera work estimation unit 18 determines whether or not the number of pattern numbers registered in the pattern table is 1, as in step 150 described above. If the determination in step 170 is denied, the process proceeds to step 172.

In step 172, the camera work estimation unit 18 sets the first direction to the i direction defined in the camera work pattern information 22 for each pattern registered in the pattern table in the same manner as in step 152 described above. It is determined whether or not all the directions D1 to Di match. Then, the camera work estimation unit 18 deletes all the pattern numbers from the pattern table for the patterns in which the first direction to the i direction do not match the approximate directions D1 to Di. Further, when there is not one pattern registered in the pattern table, the pattern number having no priority identifier given to each pattern shown in FIG. 5 is deleted from the pattern table. For example, as shown in FIG. 25, it is assumed that images I1'to image I12' are photographed in the camera work pattern 2 of FIG. In this case, when the processing from the image I1'to the image I5' is completed and the image I6' becomes the processed image, the pattern 2 and the pattern 5 are registered in the pattern table before the step 172 is executed. There is. In step 172, since the priority identifier of the pattern 2 shown in FIG. 5 is present and the priority identifier of the pattern 5 is absent, the camera work estimation unit 18 deletes the pattern 5 from the pattern table.

In step 174, the positional relationship specifying trial unit 16 increments i and j by 1 and sets the number of images L (j) to 1. Further, the i-th direction of the camera work pattern remaining in the pattern table is read from the table of FIG. 5 and set to the approximate direction Di.

In step 176, the positional relationship specifying trial unit 16 sets the image number cn of the processed image in the position unknown table for registering the information of the image whose positional relationship with the other image is unknown, as in step 126 described above. sign up.

In step 178, the positional relationship identification trial unit 16 determines whether or not the main shooting direction of the pattern registered in the pattern table is constant, so that the registered patterns are 1, 3, 6, 8, and 11. , 13, 16 and 18 are determined. If the determination in step 178 is affirmed, the process proceeds to step 180. If the determination in step 178 is denied, the process proceeds to step 182.

In step 180, the positional relationship specifying trial unit 16 performs the same matching as in step 122 described above for the processed image cn and the comparison image cn-L (j) -L (j-1) -1. For example, when the processed image is I8 shown in FIG. 8, cn = 8, j = 3, L (3) = 2, L (2) = 1, and the comparison image is I4, so matching I8 and I4 is performed. Do.

In step 182, the positional relationship identification trial unit 16 performs the same matching as in step 122 described above with the processed image cn and the comparison image cn-L (j) -L (j-1) -L (j-2) -1. Do about. For example, when the processed image is I8'shown in FIG. 25, cn = 8, j = 3, L (3) = 2, L (2) = 1, L (1) = 3, and the comparison image is I1'. Therefore, I8'and I1' are matched.

In step 184, the positional relationship specifying trial unit 16 determines whether or not the processed image and the comparison image are continuous, as in step 124 described above. If the determination in step 184 is affirmed, the process proceeds to step 186. If the determination in step 184 is denied, the process proceeds to step 188.

In step 186, the positional relationship specifying trial unit 16 increases the number of images L (j) by 1. Further, the relative coordinates (Xcn, Ycn) of the processed image with the comparison image as the origin are stored in the memory 54 or the like. Further, the image number of the comparison image corresponding to the origin of the relative coordinates of the processed image is set in the reference image REFcn, and the reference image REFcn and the number of images L (j) are also stored in the memory 54 or the like. In step 188, the positional relationship specifying trial unit 16 increases the number of images L (j) by 1 and stores it in the memory 54 or the like. In step 190, the positional relationship specifying trial unit 16 sets the image number cn of the processed image in the position unknown table for registering the information of the image whose positional relationship with other images is unknown, as in step 126 described above. sign up.

When the above-described image relative position estimation process 2 is completed, the process proceeds to step 150 of the image relative position estimation process (FIG. 9). Therefore, in the image relative position estimation process (FIG. 9), step 144 or step 146 or the image relative position estimation process 2 described above is selectively executed for the images after the image number = 2. As a result, the plurality of captured images are an image in which the positional relationship with the other image can be specified, that is, an image in which the reference image REFcn and relative coordinates (Xcn, Ycn) are stored in the memory 54 or the like, and the other image. The image is discriminated from the image registered in the unknown position table because the positional relationship with the image cannot be specified.

When the image relative position estimation process is completed, the process proceeds to step 108 of the image composition process (FIG. 7), and in step 108, the positional relationship determination unit 24 and the image composition unit 26 perform the composite image generation process. The details of this composite image generation process will be described with reference to FIG.

In step 200 of the composite image generation process, the image composite unit 26 sets 1 for the image number cn of the processed image. In the next step 202, the image synthesizing unit 26 determines whether or not the image number cn of the processed image is registered in the position unknown table. If the determination in step 202 is affirmed, the process proceeds to step 204.

In step 204, the image synthesizing unit 26 increments the image number cn by 1 and returns to step 202. As a result, steps 202 and 204 are repeated until the determination in step 202 is denied. Therefore, when an image having an image number = 1 is registered in the unknown position table, the image number cn is incremented by 1 until an image not registered in the unknown position table appears.

When an image not registered in the position unknown table appears, the determination in step 202 is denied and the process proceeds to step 206. At the time when the composite image generation process is performed, the number of pattern numbers registered in the pattern table is 1, and the camera work estimation unit 18 has completed the estimation of the camera work pattern. Therefore, in step 206, the image synthesizing unit 26 expands the processed image data to the storage position corresponding to the estimated start point of the camera work pattern in the composite image storage area reserved in advance in the memory 54. And place it. For example, if the estimated camera work pattern is pattern 1, the processed image data is expanded and arranged at a position corresponding to the upper left corner, which is the starting point of pattern 1.

In the next step 208, the image synthesizing unit 26 increments the image number cn by 1, and in step 210, the image synthesizing unit 26 determines whether or not the image number cn of the processed image is registered in the position unknown table. .. If the determination in step 210 is affirmed, the process returns to step 208, and steps 208 and 210 are repeated until the determination in step 210 is affirmed. As a result, the image number cn is incremented by 1 until an image that is not registered in the unknown position table appears.

When an image that is not registered in the position unknown table appears, the determination in step 210 is denied and the process proceeds to step 212. In step 212, the image synthesizing unit 26 refers to the reference image REFcn and the relative coordinates (Xcn, Ycn) corresponding to the processed image cn stored in the memory 54. Further, the image synthesizing unit 26 stores the XY coordinates of the positions corresponding to the relative coordinates (Xcn, Ycn) with the reference image REFcn already expanded and arranged in the composite image storage area as the origin, and stores the composite image of the reference image REFcn. Calculate based on the XY coordinates in the area. Then, the image synthesizing unit 26 expands and arranges the data of the processed image cn at a position corresponding to the calculated XY coordinates in the composite image storage area.

In the next step 214, the image synthesizing unit 26 determines whether or not the image number cn has reached the total number of images acquired by the image acquisition unit 12. If the determination in step 214 is denied, the process returns to step 208, and steps 208 to 214 are repeated until the determination in step 214 is affirmed. As a result, as shown in FIG. 20 as an example, among the images I1 to I12, each image excluding the images I6 and I7 having poor image features and the image numbers registered in the position unknown table is the reference image REFcn and the relative coordinates. Based on (Xcn, Ycn), they are expanded and arranged in the composite image storage area.

When the image number cn reaches the total number of images, the determination in step 214 is affirmed and the process proceeds to step 216. In step 216, the positional relationship determination unit 24 determines whether or not the position unknown table is empty. If the determination in step 216 is denied, the process proceeds to step 218, and in step 218, the positional relationship determination unit 24 selects and selects the smallest image number cn from the image numbers registered in the position unknown table. Extract the image number cn from the unknown position table.

In the next step 220, the positional relationship determination unit 24 moves up and down adjacent to the processed image cn based on the array of the camera work pattern information 22 and the number of images L (j) of the only pattern number registered in the pattern table. Recognize the image numbers of the images in the left and right four directions. Each element of the array of the number of images L (j) is the value of the number of images-1 of each image group taken until the approximate direction of movement of the shooting range changes in the camera work performed by the worker 32. It is set. Further, the approximate direction of movement of the shooting range when the images of the individual image groups are shot is defined in the camera work pattern information 22. Therefore, from these information, for example, if the pattern of the camera work is pattern 1, it can be recognized that the images in the four directions of up, down, left, and right adjacent to the image I6 shown in FIG. 20 are the images I2, I5, I7, and I10.

In step 222, the positional relationship determination unit 24 extracts an image already arranged in the composite image storage area from the images in the four directions of up, down, left, and right adjacent to the processed image cn, whose image numbers are recognized in step 220. For example, with respect to the image I6 shown in FIG. 20, the images I2, I5, I10 are arranged in the composite image storage area from the images I2, I5, I7, and I10 in the four directions of up, down, left, and right adjacent to the image I6. Is extracted.

In the next step 224, the positional relationship determination unit 24 determines the position of the processed image cn based on the approximate direction of the processed image cn with respect to the arranged image extracted in step 222 and the position of the image in the arranged composite image storage area. The position (XY coordinates) in the composite image storage area is determined. The position of the processed image can be determined by, for example, the following processing.

That is, first, the image extracted in step 222 and arranged in the composite image storage area adjacent to the processed image cn is adjacent to the processed image cn in the X direction (the approximate direction is "left" or "right"). ) Images are classified into images that are adjacent to the Y direction (the approximate direction is "up" or "down"). Next, the Y coordinate value in the composite image storage area of the processed image cn is determined from the Y coordinate value in the composite image storage area of the image adjacent to the processed image cn in the X direction. For example, if the number of arranged images adjacent to the processed image cn in the X direction is one, the Y coordinate value of the image is set to the Y coordinate value in the composite image storage area of the processed image cn. You may. Further, for example, if the number of arranged images adjacent to the processed image cn in the X direction is two, the average value of the Y coordinate values of the two images is set in the composite image storage area of the processed image cn. It may be set to the Y coordinate value in.

Further, the X coordinate value in the composite image storage area of the processed image cn is determined from the X coordinate value in the composite image storage area of the image adjacent to the processed image cn in the Y direction. For example, if the number of arranged images adjacent to the processed image cn in the Y direction is one, the X coordinate value of the image is set to the X coordinate value in the composite image storage area of the processed image cn. You may. Further, for example, if the number of arranged images adjacent to the processed image cn in the Y direction is two, the average value of the X coordinate values of the two images is set in the composite image storage area of the processed image cn. It may be set to the X coordinate value in.

Note that FIG. 21 shows an example in which the position of the image I6 shown in FIG. 20 is determined from the coordinate values of the images I2 and I5. In this way, if there is one or more images adjacent to the processed image cn in the X direction and one or more images adjacent to the Y direction, the position of the processed image cn can be determined. The position of the processed image cn may be determined from the individual images.

In the next step 226, the image synthesizing unit 26 expands and arranges the data of the processed image cn at a position in the composite image storage area of the processed image cn determined in step 224. When step 226 is performed, the process returns to step 216, and steps 216 to 226 are repeated until the determination in step 216 is affirmed. As a result, the images whose image numbers are registered in the position unknown table are expanded and arranged in the composite image storage area in ascending order of the image numbers. Then, when the position unknown table becomes empty and the determination in step 216 is affirmed, as shown in FIG. 22 as an example, a composite image in which all the images acquired by the image acquisition unit 12 are combined is generated. To. The composite image is one image in which the entire object 30 is the subject.

When the determination in step 216 is affirmed, the composite image generation process is terminated, and the process proceeds to step 110 of the image composite process (FIG. 7). In step 110, the image synthesizing unit 26 reads out the information about the crack detected by the crack detecting unit 14 from the memory 54 or the like. Then, the image synthesizing unit 26 generates a first output image in which the cracks are highlighted on the composite image from the composite image generated by the composite image generation process (FIG. 19) based on the acquired information on the cracks. FIG. 23 shows an example of the first output image generated by the image synthesizing unit 26.

In the next step 112, the image synthesizing unit 26 selects an image in which the crack is detected by the crack detecting unit 14 from the plurality of images acquired by the image acquiring unit 12 based on the information regarding the read crack. Then, the image synthesizing unit 26 highlights the cracks on the selected image (the image in which the cracks are detected) based on the information about the cracks, and adds the detailed information of the cracks to the second output image. Generate. FIG. 23 shows an example of the second output image generated by the image synthesizing unit 26.

In step 114, the image output unit 28 outputs the first output image and the second output image generated by the image composition unit 26, for example, by displaying them on the display unit 60, and ends the image composition process.

By referring to the first output image output by the image output unit 28, the operator can easily grasp the number of cracks, the approximate position, and the like in the entire object 30. In addition, the operator can easily grasp the properties of individual cracks occurring in the object 30 by referring to the second output image output by the image output unit 28.

As described above, in the present embodiment, the positional relationship identification trial unit 16 performs the following processing using each of the plurality of images of the object 30 taken for each portion as processed images. That is, the positional relationship specifying trial unit 16 is based on the image features of the processed image and the comparative image whose shooting order is continuous with the processed image, and the positional relationship specifying trial unit 16 is the object 30 appearing in the processed image with respect to the portion of the object 30 appearing in the comparative image. Try to identify the positional relationship of the parts. Further, the camera work estimation unit 18 estimates the camera work from the positional relationship of the image whose positional relationship can be specified by the positional relationship identification trial unit 16. Further, the positional relationship determination unit 24 determines the position of the portion of the object 30 in the image in which the positional relationship cannot be specified by the positional relationship identification trial unit 16 based on the camera work estimated by the camera work estimation unit 18. To do. Then, the image synthesizing unit 26 synthesizes a plurality of images. As a result, the image of the object can be obtained from the plurality of images even when the plurality of images of the object captured for each part include an image having poor image features.

Further, in the present embodiment, the positional relationship specifying trial unit 16 performs the following processing on the processed image for which the positional relationship with respect to the comparative image could not be specified. That is, the positional relationship specifying trial unit 16 has a positional relationship in which the portion of the object 30 captured in the image is adjacent to the portion of the object 30 captured in the processed image in the sub-shooting direction that intersects the main shooting direction of the camera work. Select the possible image from a plurality of images. Then, the positional relationship identification trial unit 16 attempts to specify the positional relationship of the processed image with respect to the selected image. As a result, even when the image features of one of the processed image and the comparison image in which the shooting order is continuous are poor, or the shooting ranges of the two images on the object are separated, the position is based on the image features. The number of images for which relationships are specified can be increased, and the accuracy of image composition is improved.

Further, in the present embodiment, the camera work estimation unit 18 can select a pattern that does not match the positional relationship identified by the positional relationship identification trial unit 16 among the plurality of patterns of the camera work stored in advance in the storage unit 20. Estimate camerawork by excluding it from. As a result, the process of estimating the camera work can be simplified as compared with the case where the positional relationship specifying trial unit 16 generates information representing the pattern from the identified positional relationship.

In the above description, a mode of selecting two images having a continuous shooting order from a plurality of images acquired by the image acquisition unit 12 in ascending order of the shooting order has been described, but the present invention is not limited to this, and the shooting order is not limited to this. Two images in which are consecutive may be selected in descending order of shooting order.

Further, in the above description, the mode in which the camera work estimation unit 18 estimates the camera work in parallel with the identification of the positional relationship of the image by the positional relationship identification trial unit 16 has been described, but the present invention is not limited to this. For example, after the positional relationship identification trial unit 16 completes the identification of the image positional relationship, the camera work estimation unit 18 determines the positional relationship of all the images for which the positional relationship identification trial unit 16 has been able to specify the positional relationship. The camera work may be estimated.

Further, in the above description, the approximate directions are set to four directions of up, down, left and right, and the mode of expressing the pattern of the camera work in these four directions has been described, but the present invention is not limited to this. For example, the second direction of the pattern 1 shown in FIG. 4 is "down" according to FIG. 5, but to be exact, it is "the beginning of the next line (diagonally lower left)", and the second direction of the pattern 6 is According to FIG. 5, it is "upper", but to be exact, it is "the beginning of the previous line (diagonally above left)". In this way, the general direction may be further subdivided. In this case, for example, the second direction is different between the pattern 1 and the pattern 2, and the estimation of the camera work can be completed at an earlier stage.

In addition, the embodiment in which the image composition program 70, which is an example of the image composition program according to the present invention, is stored (installed) in the storage unit 56 in advance has been described above. However, the present invention is not limited to this, and the image synthesis program according to the present invention can also be provided in a form recorded on a recording medium such as a CD-ROM, a DVD-ROM, or a memory card.

All documents, patent applications and technical standards described herein are to the same extent as if the individual documents, patent applications and technical standards were specifically and individually stated to be incorporated by reference. Incorporated by reference in the book.

Regarding the above embodiments, the following additional notes will be further disclosed.

(Appendix 1)
Of the plurality of images in which the object is photographed for each portion, the processing for the portion of the object reflected in the comparative image is based on the image features of the processed image and the comparative image in which the processed image and the imaging order are continuous. Identify the positional relationship of the part of the object shown in the image,
The camera work is estimated from the positional relationship of the identified processed image, and the camera work is estimated.
An image compositing program that causes a computer to perform a process of synthesizing a plurality of images by determining the position of a portion of the object appearing in an image whose positional relationship cannot be specified based on the estimated camera work.

(Appendix 2)
With respect to the processed image whose positional relationship with respect to the comparative image could not be specified, the portion of the object reflected in the image intersects the main shooting direction of the camera work with respect to the portion of the object reflected in the processed image. The image composition program according to Appendix 1, wherein the image that may have a positional relationship adjacent to the shooting direction is selected from the plurality of images, and an attempt is made to specify the positional relationship of the processed image with respect to the selected image.

(Appendix 3)
The image composition program according to Appendix 1 or Appendix 2 for estimating the camera work by excluding from the candidates a pattern that does not match the identified positional relationship among the plurality of patterns of the camera work stored in the storage unit in advance. ..

(Appendix 4)
Of the plurality of images in which the object is photographed for each portion, the processing for the portion of the object reflected in the comparative image is based on the image features of the processed image and the comparative image in which the processed image and the imaging order are continuous. Identify the positional relationship of the part of the object shown in the image,
The camera work is estimated from the positional relationship of the identified processed image, and the camera work is estimated.
An image compositing method in which a computer executes a process of determining the position of a portion of an object appearing in an image whose positional relationship cannot be specified based on the estimated camera work and compositing the plurality of images.

(Appendix 5)
With respect to the processed image whose positional relationship with respect to the comparative image could not be specified, the portion of the object reflected in the image intersects the main shooting direction of the camera work with respect to the portion of the object reflected in the processed image. The image composition method according to Appendix 4, wherein the image that may have a positional relationship adjacent to the shooting direction is selected from the plurality of images, and an attempt is made to specify the positional relationship of the processed image with respect to the selected image.

(Appendix 6)
The image composition method according to Appendix 4 or Appendix 5, wherein the camera work is estimated by excluding from the candidates a pattern that does not match the identified positional relationship among the plurality of patterns of the camera work stored in the storage unit in advance. ..

(Appendix 7)
Of the plurality of images in which the object is photographed for each portion, the processing for the portion of the object reflected in the comparative image is based on the image features of the processed image and the comparative image in which the processed image and the imaging order are continuous. A specific part that specifies the positional relationship of the part of the object in the image,
An estimation unit that estimates camera work from the positional relationship of the identified processed image,
Based on the estimated camera work, a determination unit that determines the position of a portion of the object appearing in an image whose positional relationship cannot be specified and synthesizes the plurality of images, and a determination unit.
Image synthesizer including.

(Appendix 8)
With respect to the processed image whose positional relationship with respect to the comparative image could not be specified, the specific portion takes a main image of the camera work with respect to the portion of the object reflected in the processed image. Addendum 7: The image is selected from the plurality of images, which may have a positional relationship adjacent to the sub-shooting direction that intersects the direction, and an attempt is made to specify the positional relationship of the processed image with respect to the selected image. Image synthesizer.

(Appendix 9)
The estimation unit estimates the camera work by excluding from the candidates a pattern that does not match the identified positional relationship among the plurality of patterns of the camera work stored in the storage unit in advance. The image synthesizer described.

10 Image synthesizer 16 Positional relationship identification trial unit 18 Camera work estimation unit 20 Storage unit 24 Positional relationship determination unit 26 Image synthesizer unit 30 Object 32 Worker 34 Imaging device 50 Computer 52 CPU
54 Memory 56 Storage 70 Image composition program

Claims (5)

Of the plurality of images in which the object is photographed for each portion, the processing for the portion of the object reflected in the comparative image is based on the image features of the processed image and the comparative image in which the processed image and the imaging order are continuous. Identify the positional relationship of the part of the object shown in the image,
The camera work is estimated from the positional relationship of the identified processed image, and the camera work is estimated.
An image compositing program that causes a computer to perform a process of synthesizing a plurality of images by determining the position of a portion of the object appearing in an image whose positional relationship cannot be specified based on the estimated camera work. With respect to the processed image whose positional relationship with respect to the comparative image could not be specified, the portion of the object reflected in the image intersects the main shooting direction of the camera work with respect to the portion of the object reflected in the processed image. The image composition program according to claim 1, wherein the image that may have a positional relationship adjacent to the shooting direction is selected from the plurality of images, and an attempt is made to specify the positional relationship of the processed image with respect to the selected image. The image according to claim 1 or 2, wherein the camera work is estimated by excluding from the candidates a pattern that does not match the identified positional relationship among the plurality of patterns of the camera work stored in the storage unit in advance. Synthesis program. Of the plurality of images in which the object is photographed for each portion, the processing for the portion of the object reflected in the comparative image is based on the image features of the processed image and the comparative image in which the processed image and the imaging order are continuous. Identify the positional relationship of the part of the object shown in the image,
The camera work is estimated from the positional relationship of the identified processed image, and the camera work is estimated.
An image compositing method in which a computer executes a process of determining the position of a portion of an object appearing in an image whose positional relationship cannot be specified based on the estimated camera work and compositing the plurality of images. Of the plurality of images in which the object is photographed for each portion, the processing for the portion of the object reflected in the comparative image is based on the image features of the processed image and the comparative image in which the processed image and the imaging order are continuous. A specific part that specifies the positional relationship of the part of the object in the image,
An estimation unit that estimates camera work from the positional relationship of the identified processed image,
Based on the estimated camera work, a determination unit that determines the position of a portion of the object appearing in an image whose positional relationship cannot be specified and synthesizes the plurality of images, and a determination unit.
Image synthesizer including.

Images