JP4453716B2 - Still image synthesizing apparatus and still image synthesizing method from moving image - Google Patents

Description

  The present invention relates to a still image synthesizing apparatus that extracts an image of a moving subject portion from a frame image in a moving image and combines it with a still image, and a still image synthesizing method from a moving image.

  Conventionally, some digital cameras have a continuous shooting mode in which a subject is continuously shot a plurality of times by one shooting operation. Such a shooting mode can record an image of a moving subject in time series, and is used, for example, when performing a form analysis such as a golf swing or a baseball pitching operation.

  Furthermore, from the plurality of images recorded in the continuous shooting mode described above, the subject portion that has moved for each image is cut out, and the images of the plurality of subject portions that have been cut out are combined into a single image, thereby moving the subject movement. A technique for enabling visual recognition has been proposed by the present applicant (see Patent Document 1 below).

JP-A-11-331693

However, when a partial image extracted from each frame image is combined with a still image, there is a problem in that it is not possible to compensate for a positional shift caused by camera shake during shooting.

The present invention has been made in view of such conventional problems, and eliminates the influence of camera shake at the time of shooting from a finally obtained still image and can improve the quality of a still image to be generated. An object of the present invention is to provide an image composition device and a method for synthesizing a still image from a moving image.

In order to solve the above-mentioned problem, in the invention of claim 1, an acquisition means for acquiring a moving image and a motion vector of each part in a frame image included in the moving image acquired by the acquisition means. a selecting means for selecting a partial image in the frame image, and combining means for combining the partial image selected by the selection means to the still image, based on the motion vector of each part in the same frame image, for their part A calculating means for calculating a frame motion vector indicating an overall movement tendency, and a combining position for the still image of the partial image combined with the still image by the combining means based on the frame motion vector calculated by the calculating means. And correction means for correcting .

In such a configuration, when a partial image extracted from each frame image is combined with a still image, it is possible to compensate for a positional shift caused by camera shake at the time of shooting.

According to a second aspect of the present invention, the selection unit includes a motion vector determination unit that determines whether or not there is a portion in the frame image where the motion vector is larger than a predetermined threshold. The image of the part for which the motion vector is determined to be larger than the predetermined threshold by the determination means is selected as the partial image.

  In such a configuration, it is possible to extract only a partial image suitable for synthesis with a still image.

In the invention of claim 3, the selecting means further selects a partial image of a part adjacent to the part of the image determined by the motion vector determining means to have a motion vector larger than a predetermined threshold. To choose as.

  In such a configuration, it is possible to reduce the rate at which partial images that should be extracted and combined in each frame image are missing.

In the invention according to claim 4, the selection means further includes selection condition determination means for determining whether or not the adjacent partial images satisfy a predetermined selection condition, and the selection condition determination means The adjacent partial images determined to satisfy the predetermined selection condition are selected.

  In such a configuration, a partial image to be synthesized with a still image can be accurately extracted from each frame image.

Further, in the invention of claim 5, there is provided contour extracting means for extracting the contour of the partial image in the frame image, and the selection condition determining means has a motion vector larger than a predetermined threshold by the contour extracting means. It is determined whether or not the degree of continuity between the contour extracted from the partial image and the contour extracted from the adjacent partial image by the contour extracting means is greater than a predetermined threshold value.

In such a configuration, a partial image to be synthesized can be extracted from each frame image with higher accuracy.

The invention according to claim 6 further comprises a designation unit that designates an arbitrary frame image among a plurality of frame images constituting the moving image acquired by the acquisition unit, and the synthesis unit includes the selection unit. The partial image selected by the above is synthesized with the frame image designated by the designation means.

According to a seventh aspect of the invention, there is provided image output means for displaying or printing out the synthesized still image generated by the synthesizing means.

In the invention according to claim 8, the acquisition means includes a memory for storing a moving image, and acquires the moving image stored in the memory.

In the invention of claim 9, the acquisition means includes an imaging means for imaging a subject and outputting a moving image, and acquires the moving image output from the imaging means.

According to the invention of claim 10, a partial image in the frame image is selected based on a step of acquiring a moving image and a motion vector of each portion in the frame image included in the acquired moving image. A step of synthesizing the selected partial image into a still image, and a step of calculating a frame motion vector indicating a tendency of the overall motion of those portions based on the motion vector of each portion in the same frame image And a method for synthesizing a still image from a moving image, comprising: correcting a combining position of the partial image combined with the still image with respect to the still image based on the calculated frame motion vector.

Further, in the invention of claim 11, based on the processing of acquiring the moving image in the computer of the still image synthesizing apparatus and the motion vector of each part in the frame image included in the acquired moving image, A process of selecting a partial image in the frame image, a process of synthesizing the selected partial image with a still image, and an overall movement tendency of those parts based on a motion vector of each part in the same frame image A still image synthesis program for executing a process for calculating a frame motion vector indicating the position and a process for correcting a synthesis position of a partial image to be synthesized with the still image based on the calculated frame motion vector It was.

As described above, in the present invention, when a partial image extracted from each frame image is combined with a still image, it is possible to compensate for a positional shift caused by camera shake during shooting. Therefore, it is possible to eliminate the influence of camera shake at the time of shooting from the finally obtained still image, and it is possible to improve the quality of the generated still image.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is a block diagram showing a still image synthesizing apparatus 1 common to each embodiment of the present invention. The still image synthesizing apparatus 1 is a still image that can visually grasp the motion of a subject from moving image data compliant with the MPEG (Motion Picture Experts Group) standard, that is, a moving image encoded using an inter-frame predictive encoding technique. The image data capturing unit 2, the work memory 3, the operation unit 4, the image processing unit 5, the monitor 8, and the printer 9 are provided.

  Here, an outline of a moving image recorded by the MPEG system will be described. That is, in the MPEG system, as shown in FIG. 11, a series of frame images F1 to Fn, an I picture (Intra Picture, intra-frame encoded image), and a P picture (Predictive Picture, forward prediction) are included. (Encoded image) and B picture (Bidirectionally Picture, bi-directional predictive encoded image) are mixed. The I picture is a complete picture recorded by intraframe coding, while the P and B pictures are pictures recorded by interframe predictive coding and are divided into a predetermined number as shown in FIG. Each of the macroblocks is represented by a motion vector (MV) and a prediction error, and together with the other frame images (the past frame image in the P picture, the past or the future in the B picture). Frame image) is restored using the restoration prediction image. The motion vector is data indicating the direction and distance that the subject has moved from the past (or future) frame, and the prediction error is the difference information of the pixel value from the past (or future) frame. This is data quantized by a DCT (Discrete Course Transform) coefficient. That is, the P picture and the B picture are images that cannot be generated unless a series of reproduction processes starting from one of the I pictures.

  On the other hand, the image data capturing unit 2 described above is a part that captures moving image data having the above-described configuration. For example, a moving image encoder provided in a digital camera or a digital video camera, a recording medium such as a hard disk in which moving image data is stored, A disk drive to which a video CD in which moving image data is stored is mounted. The work memory 3 is a DRAM or the like that stores each frame data of a moving image and stores still image data created by a method described later based on the frame data. The operation unit 4 includes various operation keys as a user interface, and includes an operation key for instructing from which part of the moving image a still image is to be synthesized.

  The image processing unit 5 includes a decoding unit 6 that decodes moving image data and an image combining unit 7 that combines still images based on the decoded data. FIG. 2 is a block diagram showing details of the image processing unit 5. The decoding unit 6 includes first and second variable length decoding units 6a and 6b, an inverse quantization unit 6c, an inverse DCT unit 6d, A motion compensation unit 6e and a frame memory 6f are provided. The image composition unit 7 includes an adoption determination unit 7a and a composition processing unit 7b, and the adoption determination unit 7a functions as a selection unit of the present invention. The composition processing unit 7b includes a still buffer 71 used in the process of generating still image data and functions as the composition means of the present invention.

  The monitor 8 displays the moving image data decoded by the image processing unit 5 and the synthesized still image, and the printer 9 prints the synthesized still image.

  Next, the operation of the still image synthesizing apparatus 1 having the above configuration will be described. FIG. 3 is a flowchart showing an operation when a still image is synthesized from an arbitrary recorded moving image, for example.

  The still image synthesizing apparatus 1 reads the designated moving image data from the image data capturing unit 2 (step S1), and from the plurality of frame images F1 to Fn (FIG. 11) constituting the moving image, the user can select a desired start frame. , Base frame and end frame are designated (step S2). The start frame is the first frame to be processed when combining still images, the end frame is the last frame to be processed, and the base frame is the partial image included in the frame between the start frame and the end frame. It is a frame image that is the original (background) when combining. The type of frame image to be designated as the base frame may be any of I, P, and B pictures, and any still image or blank (a solid color image recorded separately from the moving image). ) May be specified.

  The designation of each frame is performed, for example, by selecting a desired frame image while reproducing a moving image from the top and displaying it on the monitor 8. At that time, in the image processing unit 5, the decoding unit 6 restores each frame image by the following operation. In the following description, for the sake of convenience, it is assumed that a plurality of frame images F1 to Fn constituting a moving image are only an I picture and a P picture.

  First, when restoring the first frame image F1, that is, an I picture, for each block, decoding of the DTC coefficient by the first variable length decoding unit 6a, inverse quantization processing by the inverse quantization unit 6c, and inverse DCT unit 6d An inverse DCT operation is sequentially performed to restore an image of the target block (hereinafter referred to as a partial image). And while outputting it to the work memory 3 mentioned above, it accumulate | stores also in the frame memory 6f. By performing this for all blocks, the frame image F1 is generated in the work memory 3 and the frame memory 6f.

  Further, when restoring the second and subsequent frame images, that is, P pictures, after separating the DTC coefficient and the motion vector MV (see FIG. 12), the first and second variable length decoding units 6a and 6b respectively separate them. Decoding is performed, and inverse quantization processing and inverse DCT calculation are sequentially performed on the decoded DTC coefficients to obtain prediction error image data. Further, the motion compensation unit 6e sets the image stored in the frame memory 6f as a predicted image for restoration, from which the image data of the block portion represented by the decoded motion vector MV is extracted, and is used as the prediction error. The partial image is restored by adding it to the image data. And while outputting it to the work memory 3 mentioned above, it accumulate | stores also in the frame memory 6f. By performing this for all the blocks, a frame image is generated in the work memory 3, and the same image is updated as a new predicted image for restoration in the frame memory 6f.

  On the other hand, when the above-mentioned start frame, base frame, and end frame are designated, the still picture data of the designated base frame is stored in the still buffer 71 of the composition processing unit 7b (step S3). As described above, when a moving image is reproduced from the head and displayed on the monitor 8 to designate a base frame, it may be stored at that time.

  Subsequently, a target frame (initially a start frame) in the moving image is selected (step S4), a target block to be restored in that frame is further selected (step S5), and the partial image is reproduced as the above-described normal moving image reproduction. Restore in the same way as time. In the meantime, the adoption determining unit 7a extracts the motion vector MV (step S6), and determines whether it is larger than a predetermined threshold (step S7). If the motion vector MV is larger than the threshold value (YES in step S7), the restored partial image is synthesized with the base frame image (still image) stored in the still buffer 71 in the synthesis processing unit 7b (step S7). S8).

  If the motion vector MV is equal to or less than the threshold (NO in step S7), the above synthesis process is skipped, and the processes in steps S6 to S8 are performed for the next target block. When the processing for all the blocks is finished (YES in step S9), the next frame is selected as the target frame (step S4), and then the processing of steps S5 to S9 is repeated. That is, only the partial image extracted from each frame image and the image of the subject portion that has moved more than a certain amount compared to the previous frame image is included in the image stored in the still buffer 71. Are combined in decoding order.

  When the processing for all the frames up to the end frame is completed (YES in step S10), at that time, the synthetic image data stored in the still buffer 71, that is, the movement of the subject can be visually grasped. The data of the still image that can be output is output to the work memory 3, and the synthesis process is terminated. The still image output to the work memory 3 is then displayed on the monitor 8 and printed by the printer 9 in response to a user instruction.

  As described above, in the present embodiment, when generating a still image from which a motion of a subject can be visually grasped from a moving image, from a plurality of frame images to be processed from a start frame to an end frame, In order to selectively extract only necessary partial images based on motion vectors and combine them with still images, compare partial frame images to identify partial images of subject parts that have moved, and specify the specified partial images separately Compared to the case of cutting out, the processing is not wasteful. Therefore, it is possible to efficiently generate a still image.

  In the present embodiment, the extraction process of partial images (images for each block) and the synthesis process to a still image are performed in parallel with the restoration (decoding) process of a plurality of frame images to be processed. Therefore, as described above, the decoding unit 6 used for reproducing a moving image can be used for still image synthesis. Therefore, the configuration of the still image synthesizing apparatus 1 is simple and can be implemented easily and at low cost.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. In the first embodiment, an image of a block having a motion vector larger than the threshold is extracted from each frame image and synthesized with a still image. However, in the present embodiment, each frame constituting a moving image For example, when an image is divided and encoded into 9 × 6 blocks (macroblocks) as shown in FIG. 4, a block whose motion vector is larger than a threshold (shown in black in the figure) In addition to the image (partial image) of the block, the image (the other partial image) of the block adjacent thereto (the block indicated by hatching and hatching) is also synthesized as a still image.

  FIG. 5 is a block diagram corresponding to FIG. 2 showing the details of the image processing unit 5 in the still image synthesis apparatus of the present embodiment. In the present embodiment, an employment determination buffer 72 is provided in the above-described employment determination unit 7a. The adoption determination buffer 72 is a buffer (such as a temporary storage memory) for storing determination results as to whether or not all blocks of the target frame being decoded are blocks to be combined as still images. The adoption determination buffer 72 is configured to store a determination result for each block.

  In the present embodiment, as in the first embodiment, while decoding the target frame from the start frame to the end frame, the adoption determination buffer 72 causes the motion vector to be lower than the threshold value for each target block. Whether the value is larger than the threshold value is determined. If the value is larger than the threshold value, an adoption flag indicating that the image of the block is to be synthesized is stored in the adoption determination buffer 72. At the same time, the adoption flag is stored for other blocks adjacent to the block.

  FIG. 6 is an explanatory diagram showing the contents of the storage processing operation of the adoption flag. In FIG. 6, “block number” is assigned to each block in the employment determination buffer 72, and is decoded within the same frame. Numbers that match the order (from left to right, from top to bottom), “motion vector” is the motion vector for each block, “adoption determination” is the presence / absence of the adoption flag, and the reason why it is stored Show. In addition, “×” in “recruitment determination” indicates that the adoption flag is not recorded, “◎” indicates that the adoption flag is recorded due to the motion vector in the block being larger than the threshold, “◯” indicates that the adoption flag is recorded for the block because the motion vector of another block adjacent to the block is larger than the threshold.

  The illustrated example shows a case where the threshold value of the motion vector is, for example, (5, 5). In this case, the motion vector of the block with the block numbers 21, 22, 24 is (2, 1) Since the motion vector of the block of block number 23 (here, the block shown in black in FIG. 4) is (10, −12), the above determination and determination storage processing is repeatedly performed. In the meantime, the adoption determination of the block numbers 21 and 22 is “x” until the block of the block number 22 becomes the decoding target, but the adoption determination of the block of the block number 23 becomes the decoding target. Becomes “◎”, and at the same time, the adoption determination of the blocks with the block numbers 22 and 24 adjacent to it becomes “◯”. Although not shown in FIG. 6, the adoption determination of the blocks having the block numbers 13 to 15 and 31 to 33 is also “◯”.

  When the above determination and storage of the determination are completed for all the blocks, the adoption flag is stored in the adoption determination buffer 72 at that time (the adoption determination is “◎” or “◯” in FIG. 6). ) Block information is sent from the adoption determination unit 7a to the synthesis processing unit 7b. Further, in response to this, the composition processing unit 7b reads a plurality of partial images corresponding to these blocks from the frame memory 6f and sequentially composes them into still images stored in the internal still buffer 71.

  Thereafter, the same processing as described above is performed for each frame image while sequentially switching the target frames. When all the frames up to the end frame have been processed, the synthesized image data stored in the still buffer 71, that is, the movement of the subject is visually grasped at that time, as in the first embodiment. Still image data that can be output is output to the work memory 3, and the synthesis process is terminated.

  Here, in the present embodiment, from each frame image from the start frame to the end frame, an image of a block having a motion vector larger than the threshold (partial image) and an image of the surrounding blocks (other partial images) are also included. Since they are extracted and synthesized as still images, the following effects can be obtained compared to the first embodiment.

  In other words, the presence or absence of motion compensation between each frame of the moving image is determined as appropriate at the time when the moving image is encoded. Therefore, the motion of all moving subject portions, that is, all blocks including the subject portion is determined. Compensation is not always done. For this reason, when only a block having a motion vector larger than the threshold value is combined with a still image, there is a partial image that is not combined although a moving subject portion is included. On the other hand, when an image of a certain block constitutes a moving subject part, an image of a block adjacent to the block often constitutes the subject part. Therefore, according to the present embodiment, it is possible to reduce the rate of missing partial images that should be originally extracted and synthesized, so that the moving subject portion is more faithfully reflected in the finally generated still image. Can be made.

  In addition, as in the first embodiment, compared with different frame images, the partial image of the subject part that has moved is identified, and the process is less wasteful than when the identified partial image is cut out separately, Still image generation can be performed efficiently, and the decoding unit 6 used for moving image reproduction can be used as it is even during still image synthesis, so the configuration of the still image synthesis apparatus 1 is simple. Yes, it can be realized easily.

(Third embodiment)
Next, a third embodiment of the present invention will be described. In the second embodiment, all the images of the blocks adjacent to the block whose motion vector is larger than the threshold value are combined with the still image. However, in this embodiment, the adjacent block having a large prediction error is used. Is synthesized into a still image.

  FIG. 7 is a block diagram corresponding to FIG. 5 showing details of the image processing unit 5 in the still image synthesizing apparatus of the present embodiment. In the present embodiment, unlike the second embodiment, the prediction error (pixel value data) restored in the inverse DCT unit 6d is input to the adoption determination unit 7a, and the adoption determination buffer 72 stores the above-described determination. In addition to the result, prediction error data is stored for each block.

  Also in the present embodiment, while decoding the target frame from the start frame to the end frame, the adoption determination unit 7a determines whether or not the motion vector is larger than the threshold value for each target block. When it is larger, the adoption flag indicating that the image of the block is to be synthesized is stored in the adoption determination buffer 72. Further, the prediction error of the target block is also stored in the adoption determination buffer 72. When the above determination and storage of determination are finished for all blocks, the prediction error stored in the block adjacent to the block in which the adoption flag is stored is determined from a predetermined threshold. A new adoption flag is stored for a larger block. Note that the magnitude of the prediction error is, for example, the total value or average value of all the pixel values in the block.

  FIG. 8 is an explanatory diagram corresponding to FIG. 6 showing the contents of the above-described storage processing operation of the adoption flag. As shown in the figure, in this embodiment, the block number 23 in which the motion vector is larger than the threshold value. Among the blocks of block numbers 22 and 24 adjacent to the block number 22, the adoption determination is “x” for the block number 22 with a small prediction error, and the block adoption determination is “◯” for the block number 24 with a large prediction error. "

  When the above determination and storage of the determination are completed for all the blocks, the adoption flag is stored in the adoption determination buffer 72 at that time (the adoption determination is “◎” or “◯” in FIG. 8). ) Block information is sent from the adoption determination unit 7a to the synthesis processing unit 7b. Further, in response to this, the composition processing unit 7b reads a plurality of partial images corresponding to these blocks from the frame memory 6f and sequentially composes them into still images stored in the internal still buffer 71. The subsequent operation is the same as that of the second embodiment.

  As described above, in the present embodiment, the prediction error is larger than the threshold value in each of the frame images from the start frame to the end frame together with the image of the block whose motion vector is larger than the threshold value. Since the large ones are extracted and the partial images are synthesized as still images, the following effects can be obtained.

  That is, as described above, when only a block having a motion vector larger than the threshold is combined with a still image, an image of a block that is not extracted and not combined even though a moving subject portion is included (part Image) exists. On the other hand, when an image of a certain block constitutes a moving subject part, the image of a block adjacent to the block often constitutes the subject part, and the probability of a block having a large prediction error. Is extremely high. Therefore, compared with the second embodiment, the extraction accuracy of the partial image to be combined is high, and the moving subject portion can be accurately combined with the still image specified as the base frame.

  Also, as in the first and second embodiments, compared to different frame images, a partial image of a subject portion that has moved is identified, and the processing is wasteful compared to a case where the identified partial image is cut out separately. Therefore, the still image can be generated efficiently, and the decoding unit 6 used for moving image reproduction can be used as it is even during still image synthesis. It is simple and can be realized easily.

(Fourth embodiment)
Next, a fourth embodiment of the present invention will be described. In the third embodiment, the adoption determination unit 7a determines whether a block adjacent to a block having a motion vector larger than the threshold can be combined on the condition that the prediction error is larger than a predetermined threshold (selection condition). In this embodiment, the determination is made by adding that the image of the block has continuity in the contour between the image of the block and the image of the block whose motion vector is larger than the threshold value. .

  FIG. 9 is a block diagram showing details of the image processing unit 5 in the still image synthesizing apparatus according to the present embodiment and corresponding to that shown in FIG. 7 in the third embodiment. Unlike the third embodiment, the present embodiment is the contour extraction means of the present invention that inputs the prediction image data for restoration stored in the frame memory 6f and extracts the contour from the input image. A contour extracting unit 7c is provided, and the contour data extracted there is output to the adoption determining unit 7a.

  Also in this embodiment, while decoding the target frame from the start frame to the end frame, the adoption determination unit 7a is to synthesize an image of a block whose motion vector is larger than the threshold for each target block. The adoption flag indicating the presence is stored in the adoption determination buffer 72. Further, when the above determination and storage of the determination flag are completed for all the blocks, at that time, among the blocks adjacent to the block in which the adoption flag is stored, the stored prediction error is determined in advance. A new adoption flag is stored for a block larger than the threshold.

  Further, in the present embodiment, the adoption determination unit 7a performs the following in each image for the block in which the adoption flag is stored and each block adjacent to the block (except for the block in which the adoption flag is stored). The contours are compared by a predetermined method in accordance with the positional relationship in the frame image, and the continuity of the contour between the two is digitized. If any of the adjacent blocks has reached the predetermined level, a new adoption flag is stored in the adoption determination buffer 72 for that block.

  Thereafter, as in the third embodiment, the information of the block in which the adoption flag is stored in the employment determination buffer 72 is sent from the employment determination unit 7a to the synthesis processing unit 7b. Are read out from the frame memory 6f and are sequentially combined with the still images stored in the internal still buffer 71.

  As described above, in the present embodiment, as a condition for determining whether or not a block adjacent to a block having a motion vector larger than the threshold value can be combined, the image of the block is a block having a motion vector larger than the threshold value. Since it has been added that the contour has a certain continuity with the image, the partial image to be synthesized can be extracted from each frame image with higher accuracy than in the third embodiment. As a result, only a moving subject portion can be reliably combined with a still image designated as a base frame.

  Also, as in the first to third embodiments, compared to different frame images, a partial image of a subject portion that has moved is specified, and the processing is wasteful compared to a case where the specified partial image is cut out separately. Therefore, the still image can be generated efficiently, and the decoding unit 6 used for moving image reproduction can be used as it is even during still image synthesis. It is simple and can be realized easily.

(Fifth embodiment)
Next, a fifth embodiment of the present invention will be described. In this embodiment, the position is corrected when a partial image extracted by the same method as in the third embodiment is combined with a still image while decoding the target frame from the start frame to the end frame. It is.

  FIG. 10 is a block diagram showing details of the image processing unit 5 in the still image synthesizing apparatus according to the present embodiment and corresponding to that shown in FIG. 7 in the third embodiment. In the present embodiment, unlike the third embodiment, the image composition unit 7 (FIG. 1) includes a frame motion vector estimation unit 7d in addition to the adoption determination unit 7a and the composition processing unit 7b, and the computing means of the present invention. And a frame motion vector accumulating unit 7e functioning as a frame motion vector estimating unit 7d. The frame motion vector estimating unit 7d receives the motion vector of the target block decoded by the second variable length decoding unit 6b.

  In this embodiment, the image composition unit 7 operates as follows while decoding the target frame. That is, the adoption determination unit 7a determines whether or not the image should be synthesized as a still image for all blocks of the same frame in the procedure described in the third embodiment, and the determination result is used as the adoption determination buffer 72. To remember. On the other hand, the frame motion vector estimation unit 7d performs statistical processing for counting the number of appearances of each input motion vector for each block, and sends the final processing result to the frame motion vector accumulation unit 7e. Output. The frame motion vector accumulating unit 7e calculates a frame motion vector indicating the overall motion tendency in each block based on the number of appearances (number of blocks) of the input motion vector for each content, and sends it to the synthesis processing unit 7b. Output.

  In the calculation of the frame motion vector, the motion vector having the highest number of appearances is equal to or more than the number of appearances and the threshold value for which the motion vector is determined (threshold different from that used for determining the block to be synthesized) The motion vector is determined as the frame motion vector if the determination condition is smaller than 0). If the determination condition is not satisfied, the content of the frame motion vector indicates no motion (0, 0) Is the operation.

  Here, when calculating the frame motion vector, if the target frame is a camera shake at the time of shooting, the content of the motion vector can be the same for the block of the subject portion such as the background that originally does not move. High nature. For example, the content of the motion vector of each block is as shown in FIG. 8, and motion vectors (2, 1) indicating a slight motion such as the blocks of block numbers 21, 22, 24 occupy a large number. This is the case. If the motion vector (2, 1) satisfies the above-described determination condition, it is output as a frame motion vector to the synthesis processing unit 7b.

  Further, when the adoption determination result of each block is sent from the adoption determination unit 7a, the composition processing unit 7b reads a plurality of partial images from the frame memory 6f according to the result and stores them in the internal still buffer 71. Composite to still images in order. At this time, in the present embodiment, the composition processing unit 7b combines the read partial image with a still image by shifting the read partial image in a predetermined direction from the original block position by the distance indicated by the frame motion vector. As a result, the positional deviation caused by the camera shake in the partial image combined with the still image is compensated. Other operations are the same as those in the third embodiment.

  As described above, in this embodiment, when a partial image extracted from each frame image is combined with a still image, it is possible to compensate for a positional shift caused by camera shake at the time of shooting. Therefore, the influence of camera shake at the time of shooting can be eliminated from the finally obtained still image, and the quality of the still image is improved.

  Further, as in the first to fourth embodiments, compared with different frame images, the partial image of the subject portion that has moved is identified, and the processing is wasteful compared to the case where the identified partial image is cut out separately. Therefore, the still image can be generated efficiently, and the decoding unit 6 used for moving image reproduction can be used as it is even during still image synthesis. It is simple and can be realized easily.

  In the present embodiment, the composition position of the partial image in the still image is corrected based on the frame motion vector at the time of synthesis of the partial image, but may be as follows. That is, the adoption determining unit 7a is provided with a buffer for storing the motion vectors of all target blocks, and the frame motion vector accumulating unit 7e is configured to output a frame motion vector to the adoption determining unit 7a. The unit 7a may be configured to perform the above-described determination processing for each block after correcting all the motion vectors stored in the buffer based on the frame motion vector in advance. Even in that case, as a result, the composite position of the partial images in the still image can be corrected based on the frame motion vector.

  In the present embodiment, the basic configuration of the third embodiment has been described. However, in the other embodiments described above, the frame motion vector estimation unit 7d, the frame motion vector accumulation unit 7e, May be added to eliminate the influence of camera shake at the time of shooting from a still image.

  In the above description, a still image synthesizing apparatus that mainly generates a still image that can visually grasp the movement of the subject and displays or prints the still image has been described. For example, the present invention can be applied to a digital camera having a moving image shooting function and a digital video camera. Further, the data processing performed by the image processing unit 5 described above can be realized without using dedicated hardware resources. Therefore, for example, the present invention can be implemented by a general-purpose personal computer and predetermined software for causing it to perform data processing similar to that of the image processing unit 5.

It is a block diagram of the still picture composition device common to each embodiment of the present invention. It is a block diagram which shows the image process part in 1st Embodiment. 4 is a flowchart showing an operation when a still image is synthesized in the embodiment. It is a figure which shows the macroblock in a frame image. It is a block diagram which shows the image process part in 2nd Embodiment. It is explanatory drawing which shows the content of the memory | storage processing operation | movement of the employment flag by the employment determination part of the embodiment. It is a block diagram which shows the image process part in 3rd Embodiment. It is explanatory drawing which shows the content of the memory | storage processing operation | movement of the employment flag by the employment determination part of the embodiment. It is a block diagram which shows the image process part in 4th Embodiment. It is a block diagram which shows the image process part in 5th Embodiment. It is a schematic diagram which shows the structure of moving image data. It is explanatory drawing which shows the content of the data which comprises a frame image.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Still image synthesizing | combining apparatus 2 Image data taking-in part 5 Image processing part 6 Decoding part 6a 1st variable length decoding part 6a 2nd variable length decoding part 6c Inverse quantization part 6d Inverse DCT part 6e Motion compensation part 6f Frame memory 7 Image composition unit 7a Adoption determination unit 7b Composition processing unit 7c Outline extraction unit 7d Frame motion vector estimation unit 7e Frame motion vector accumulation unit 8 Monitor 9 Printer

Claims (11)

An acquisition means for acquiring a moving image;
Selection means for selecting a partial image in the frame image based on a motion vector of each part in the frame image included in the moving image acquired by the acquisition means ;
Combining means for combining the partial image selected by the selecting means with a still image ;
Based on the motion vector of each part in the same frame image, a calculation means for calculating a frame motion vector indicating the overall motion tendency of those parts;
A still image synthesizing apparatus comprising: correction means for correcting a synthesis position of a partial image synthesized with a still image by the synthesis means based on a frame motion vector calculated by the calculation means . The selection means includes
Motion vector determining means for determining whether or not there is a portion in which the motion vector is larger than a predetermined threshold in the frame image;
2. The still image synthesizing apparatus according to claim 1, wherein an image of a part for which a motion vector is determined to be larger than a predetermined threshold by the motion vector determination unit is selected as a partial image. The selecting means further includes:
3. The still image synthesizing apparatus according to claim 2 , wherein an image of a portion adjacent to an image of a portion whose motion vector is determined to be larger than a predetermined threshold by the motion vector determining means is selected as a partial image. The selecting means further includes:
A selection condition determining means for determining whether or not the adjacent partial image satisfies a predetermined selection condition;
4. The still image synthesizing apparatus according to claim 3 , wherein the adjacent partial images determined to satisfy a predetermined selection condition by the selection condition determining means are selected. Contour extraction means for extracting the contour of the partial image in the frame image,
The selection condition determining means includes
The degree of continuity between the contour extracted from the image of the portion where the motion vector is larger than a predetermined threshold by the contour extracting unit and the contour extracted from the adjacent partial image by the contour extracting unit is predetermined. 5. The still image synthesis apparatus according to claim 4, wherein it is determined whether or not the threshold value is larger than the threshold value. A designating unit for designating an arbitrary frame image among a plurality of frame images constituting the moving image obtained by the obtaining unit;
The combining means, the partial image selected by the selecting means, still image synthesizing apparatus according to any one of claims 1 to 5, characterized in that the synthesis to the designated frame image by the specifying means. 7. The still image synthesizing apparatus according to claim 1, further comprising image output means for displaying or printing out the synthesized still image generated by the synthesizing means. The acquisition means includes
Including a memory for storing moving images,
8. The still image synthesizing apparatus according to claim 1, wherein a moving image stored in the memory is acquired. The acquisition means includes
Including imaging means for imaging a subject and outputting a moving image;
The still image synthesizing apparatus according to claim 1, wherein a moving image output from the imaging unit is acquired. Acquiring a moving image;
Selecting a partial image in the frame image based on a motion vector of each part in the frame image included in the acquired moving image;
Combining the selected partial image with a still image;
Calculating a frame motion vector based on a motion vector of each part in the same frame image and indicating an overall motion tendency of those parts;
Correcting a combined position of the partial image combined with the still image based on the calculated frame motion vector with respect to the still image;
A method for synthesizing a still image from a moving image. In the computer that the still image composition device has,
Processing to acquire a moving image;
A process of selecting a partial image in the frame image based on a motion vector of each part in the frame image included in the acquired moving image;
A process of combining the selected partial image with a still image;
Based on the motion vector of each part in the same frame image, a process for calculating a frame motion vector indicating the overall motion tendency of those parts;
Processing for correcting a composite position of a partial image to be combined with the still image based on the calculated frame motion vector with respect to the still image;
Still image composition program to execute.

Images