JP4513699B2 - Moving image editing apparatus, moving image editing method and program - Google Patents

Description

  The present invention relates to a technique effectively applied to an apparatus for framing a moving image.

  Due to the recent miniaturization of image pickup devices, camera shake during image pickup has become prominent in image pickup results, which is a problem. For amateurs, it is difficult to accurately frame the subject. Due to such blurring and inaccurate framing, there arises a problem that the visibility of the subject of interest is lowered.

  In order to solve such a problem, an automatic framing technique has been proposed in which the subject is held at the center of the screen during shooting. For example, a technique has been proposed in which a wide-area image is captured and a tracking image in which the subject does not go out of the frame is obtained by continuously cutting out partial images including the subject of interest (Patent Document 1). reference).

In addition, a technique for acquiring and displaying an image in which a person is placed at the center of the screen (see Patent Document 2) and a technique for capturing an image of a manually designated subject (see Patent Document 3) have been proposed. .
JP 2004-207884 A JP 2000-078564 A Japanese Patent Laid-Open No. 06-038214

  In any of the prior arts, an object of interest is to improve the visibility of an object by extracting a subject of interest from a captured image and performing framing by capturing the subject of interest at the center of the screen. In a moving image framed in this manner, the subject of interest always exists at the center of the screen, and thus visibility is improved. However, in the video by such a method, there is a problem that the position of the subject of interest does not move from the center of the screen and the feeling of dynamism is lost. Such a problem does not occur in imaging with a surveillance camera whose main purpose is to improve the visibility of the subject of interest. However, there is a problem when the purpose is to capture an everyday scene, such as imaging with a home digital video camera.

  Accordingly, an object of the present invention is to solve these problems and to provide an apparatus and a method for generating a moving image that does not lose its dynamic feeling while maintaining the visibility of the subject of interest.

  In order to solve the above problems, the present invention has the following configuration. The present invention is a moving image editing apparatus, and includes a position acquisition unit and a cutout unit. The position acquisition unit acquires the position of the object in each frame of the input moving image. The cutout means determines the position of the cutout frame in the frame based on the reference position preset in the frame and the position of the target so that the target is contained in the cutout frame, and is surrounded by the cutout frame. Reconstruct moving images by cutting out images. At this time, the cutout unit determines the position of the cutout frame so that the influence of the position of the target object on the vertical component of the cutout position is smaller than the horizontal component. The target object may be arbitrarily set, for example, a human head, face, car, ball or the like.

According to the present invention configured as described above, when a part of each frame of the moving image is cut out and a new moving image is reconstructed, the position of the cutting frame is determined by the cutting means. For this reason, since the component in the horizontal direction changes according to the position of the object, the visibility of the subject of interest (object) can be maintained. Furthermore, the vertical component is less affected by the position of the object than the horizontal component, in other words, the followability to the position of the object is low. For this reason, the vertical position of the object within the clipping frame varies greatly from frame to frame in each frame. For this reason, since the vertical position change is maintained in the cut out image, it is possible to generate a moving image that does not lose its dynamic feeling.

  The cut-out means of the present invention obtains the difference between the reference position and the position of the object, multiplies the horizontal component of the difference by the first value, and smaller than the first value for the vertical component of the difference. The position of the cutout frame may be determined by multiplying the second value and using the obtained both values and the reference position.

  The clipping means of the present invention obtains a difference between the reference position and the position of the object in a plurality of frames, a time-series horizontal component in consideration of a time-series change between frames for the horizontal component and the vertical component of the difference, and Calculate the time-series vertical component, multiply the time-series horizontal component by the first value, multiply the time-series vertical component by the second value smaller than the first value, and use both the obtained values and the reference position. The position of the cutout frame may be determined.

The time-series horizontal component and the time-series vertical component may be low-frequency components of the time-series change of the horizontal component and the vertical component .

  The present invention may be realized by a program being executed by an information processing apparatus. That is, the present invention can specify the processing executed by each of the above-described means as a program for causing the information processing apparatus to execute, or a recording medium on which the program is recorded. Further, the present invention may be specified by a method in which the information processing apparatus executes the processing executed by each of the above-described means. When the present invention is specified by the method, it does not matter whether the information processing apparatus performs only image editing or plays back (screens) while editing the image. Furthermore, the present invention may be specified by a recording medium on which a moving image edited by the above method is recorded.

  According to the present invention, it is possible to generate a moving image that maintains the visibility of the subject of interest and does not lose its dynamic feeling.

  [System Configuration] FIG. 1 is a diagram illustrating an example of functional blocks of the moving image editing apparatus 1. Hereinafter, the moving image editing apparatus 1 will be described as an example of the moving image editing apparatus. The moving image editing apparatus 1 detects / tracks a person's head as an object and uses it as a framing target. However, the moving image editing apparatus 1 detects / tracks another object (such as a car or a ball) and uses it as a framing target. It may be configured.

  The moving image editing apparatus 1 includes a CPU (Central Processing Unit), a main storage device (RAM), an auxiliary storage device, and the like connected via a bus in terms of hardware. The moving image editing apparatus 1 is loaded with various programs (OS, applications, etc.) stored in the auxiliary storage device and executed by the CPU, whereby the image input unit 2, the temporary storage unit 3, the head It functions as an apparatus including the detection unit 4, the head tracking unit 5, the cutout unit 6, and the like. The head detection unit 4, the head tracking unit 5, and the cutout unit 6 are realized by executing a program by the CPU. Moreover, the head detection unit 4, the head tracking unit 5, and the cutout unit 6 may be configured as dedicated chips. Next, each functional unit included in the moving image editing apparatus 1 will be described.

<Image Input Unit> The image input unit 2 functions as an interface for inputting moving image data to the moving image editing apparatus 1. The image input unit 2 inputs moving image data to the moving image editing apparatus 1. The image input unit 2 may be configured using any existing technique for inputting moving image data to the moving image editing apparatus 1.

  For example, moving image data may be input to the moving image editing apparatus 1 via a network (for example, a local area network or the Internet). In this case, the image input unit 2 is configured using a network interface. In addition, moving image data may be input to the moving image editing apparatus 1 from a digital video camera, a personal computer, a recording device (for example, a hard disk drive), or the like. In this case, the image input unit 2 is a standard (for example, USB (Universal Serial Bus) or SCSI (Small Computer System Interface)) that connects the digital video camera, personal computer, recording device, and the like to the moving image editing apparatus 1 so that data communication is possible. Or the like (wireless connection standard such as Bluetooth (registered trademark)). Also, moving image data recorded on a recording medium (for example, various flash memories, CD (Compact Disk), DVD (Digital Versatile Disc, Digital Video Disc)) may be input to the moving image editing apparatus 1. In this case, the image input unit 2 is configured using a device (for example, a flash memory reader, a CD drive, or a DVD drive) that reads data from a recording medium.

  In addition, the moving image editing apparatus 1 is included in various apparatuses (for example, a PDA (Personal Digital Assistant) or a mobile phone) including an image capturing apparatus such as a digital video camera or an image capturing apparatus such as a digital video camera. May be input to the moving image editing apparatus 1. In this case, the image input unit 2 is an interface for causing the moving image editing apparatus 1 to input moving image data picked up by an image pickup device such as a charge-coupled device (CCD) or a complementary metal-oxide semiconductor (CMOS) sensor. It may be configured as. Further, the image input unit 2 may be configured to be able to respond to a plurality of cases described above.

  <Temporary Storage Unit> The temporary storage unit 3 is configured using a storage device. Any specific technique such as a volatile storage device or a nonvolatile storage device may be applied to the storage device used for the temporary storage unit 3.

  The temporary storage unit 3 stores moving image data input via the image input unit 2. Data of each frame of the moving image stored in the temporary storage unit 3 is read by the head detection unit 4, the head tracking unit 5, and the cutout unit 6. The temporary storage unit 3 holds data of each frame of a moving image that is a target of the processing until at least the processing by the head detection unit 4, the head tracking unit 5, and the cutout unit 6 is completed.

  <Head Detection Unit> The head detection unit 4 reads the frame data of the moving image from the temporary storage unit 3, detects the human head from the data, and indicates the position and size of the detected head. Identify head information. The head detection unit 4 may be configured to detect the head by detecting the face by template matching using a reference template corresponding to the outline of the entire face, for example. The head detection unit 4 may be configured to detect a vertex such as a head by chroma key processing and detect the head based on the vertex. The head detection unit 4 may be configured to detect a head by detecting a region close to the skin color and detecting the region as a face. Further, the head detection unit 4 may be configured to perform learning based on a teacher signal using a neural network and detect a face-like area or a head-like area as a head. In addition, the detection processing by the head detection unit 4 may be realized by applying any existing technique.

<Head Tracking Unit> The head tracking unit 5 tracks (tracks) the head detected by the head detection unit 4 in a moving image stored in the temporary storage unit 3. Moreover, the head tracking part 5 determines the head used as the object of a tracking process according to a specific reference | standard, when the head of several people is detected from the flame | frame of a moving image. The predetermined reference is, for example, a face size, a face orientation, a face position in an image, or the like. The tracking process in the head tracking unit 5 is performed by searching for and tracking feature points included in the head (for example, feature points on the forehead, eyebrows, eyes, ears, nose, lips, etc.) around the feature points of the previous image. It can be realized by doing. Further, this tracking process may be realized by other existing technology.

  <Cutout Unit> The cutout unit 6 cuts out a frame to be output according to the tracking result (that is, the head position) of the head tracking unit 5. FIG. 2 is a diagram illustrating an example of a frame position cut out by the cutout unit 6. The reference point (X, Y) is a point serving as a reference for processing by the cutout unit 6 and may be arbitrarily set without changing with time t. In FIG. 2, the reference point is the center point of the frame. The head center point (x (t), y (t)) is a point obtained from the tracking result of the head tracking unit 5 and indicates the center position of the head of the person who is the subject at a certain time t. The center point of the head is an example indicating the position of the head of the person who is the subject, and need not be the center of the head, for example, the center of the face, the center of both eyes, the position of the nose or mouth Also good. The head center vector (Xd (t), Yd (t)) is a vector indicating the head center point from the reference point. The frame center vectors (Xs (t), Ys (t)) are vectors to be calculated by the cutout unit 6, and are vectors indicating the center position of the frame that the cutout unit 6 is to cut out from the reference point. Hereinafter, an example of how to obtain the frame center vector will be described.

  The cutout unit 6 obtains a frame center vector based on the positions of the reference point and the head center point. At this time, the clipping unit 6 obtains the frame center vector so that the difference between the frames of the Y-axis component (vertical direction component) of the frame center vector becomes small or becomes zero. Specifically, in the clipping unit 6, for example, the influence of the head center point on the Y-axis component of the frame center vector is smaller than the influence of the head center point on the X-axis component (horizontal direction component) of the frame center vector. Or the center vector of the frame can be determined so that the influence is zero.

切り出し部６は、上述した処理によってフレーム中心ベクトルを算出すると、このフレーム中心ベクトルによって示された点を中心としてフレーム位置を決定する。フレーム位置は、フレーム中心ベクトルとフレームの縦横の長さによって決定することができる。フレームの縦横の長さは、予め定められていても良いし、ユーザが設定可能でも良い。 In order to perform such processing, the cutout unit 6 obtains a frame center vector based on, for example, Equation 1. That is, the clipping unit 6 multiplies the Y-axis component of the head center vector by a value (k) smaller than the value (j) multiplied by the X-axis component of the head center vector or zero as described above. Processing can be realized.

When the cutout unit 6 calculates the frame center vector by the above-described processing, the cutout unit 6 determines the frame position around the point indicated by the frame center vector. The frame position can be determined by the frame center vector and the vertical and horizontal lengths of the frame. The vertical and horizontal lengths of the frame may be determined in advance or may be set by the user.

  The cutout unit 6 cuts out the image in the frame determined in this way, and outputs it to the moving image storage unit 7.

  <Moving Image Storage Unit> The moving image storage unit 7 is configured using a nonvolatile storage device. The moving image storage unit 7 may be provided in the moving image editing apparatus 1 or may be provided outside the moving image editing apparatus 1 as shown in FIG. The moving image storage unit 7 stores a moving image composed of frames cut out by the cutout unit 6.

  For example, when the moving image editing apparatus 1 is mounted on a digital video camera, the recording medium of the digital video camera corresponds to the moving image storage unit 7. For example, when the moving image editing apparatus 1 is mounted on a personal computer, a hard disk, a CD-R, a DVD-R, or the like corresponds to the moving image storage unit 7.

  [Operation Example] FIG. 3 is a flowchart illustrating an operation example of the moving image editing apparatus 1. When a moving image is input via the image input unit 2 (S01), data of each frame of the moving image is stored in the temporary storage unit 3. The moving image editing apparatus 1 performs the following processing for each input frame.

  First, the cutout unit 6 acquires the head center point from the processing result of the head detection unit 4 or the head tracking unit 5 (S02). Next, the clipping unit 6 calculates a head center vector from the reference point and the head center point (S03), and calculates a frame center vector using Equation 1 (S04). Then, the cutout unit 6 cuts out a frame determined by the frame center vector (S05), and outputs moving image data constituted by the frame (S06).

  [Operation / Effect] FIG. 4A is an example of a frame cut out by the conventional technique. FIG. 4B shows an example in which a moving image edited by the prior art is reproduced. FIG. 5A shows an example of a frame cut out by the cutout unit 6 of the moving image editing apparatus 1. FIG. 5B shows an example in which a moving image edited by the moving image editing apparatus 1 is reproduced.

  In the prior art, the position of the frame is determined only by the position of the person's head and is independent of the position of the reference point. Therefore, as shown in FIG. 4B, the human head is always located at the approximate center of the frame.

  On the other hand, in the moving image editing apparatus 1, the position of the frame is determined by the position of the human head (head center point) and the position of the reference point. At this time, the position in the vertical direction of the frame is less affected by the head center point than the position in the horizontal direction (X-axis direction), and is greatly affected by the reference point. And the position of the reference point does not move between the frames. For this reason, as shown in FIG. 5 (b), the human head is always positioned at the approximate center of the frame in the X-axis direction, but is not fixed at the approximate center in the Y-axis direction. Is displaced. That is, when a moving image composed of frames cut out by the moving image editing apparatus 1 is reproduced, the human head swings up and down (vertical direction) in accordance with the movement in the Y-axis direction. Therefore, the human head is always captured at the approximate center in the X-axis direction, so that visibility is maintained, and the human head swings up and down naturally in the Y-axis direction, so that the feeling of dynamism is not lost.

  When the moving image output by the moving image editing apparatus 1 is a moving image in which, for example, a figure of a person running is captured, the following effects are also obtained. When a person runs, the person's head moves up and down in small increments. When framing is performed so that the human head is always centered as in the past, the human head is clearly visible, but the background shakes up and down in small increments. End up. For this reason, not only the feeling of dynamism is lost, but also a moving image that is difficult to see due to the flickering background has been generated.

On the other hand, when framing is performed by the moving image editing apparatus 1, not the background but the human head moves up and down in small increments. For this reason, it is suppressed that a background shakes in small steps and it becomes difficult to see.

  [Modification] The moving image editing apparatus 1 may be configured not to include the head tracking unit 5. In this case, the head detection unit 4 may be configured to perform the head tracking process by repeatedly executing the detection process on the moving image stored in the temporary storage unit 3.

式２において、Ｘｄ’（ｔ）、Ｙｄ’（ｔ）は、頭部中心ベクトル（Ｘｄ（ｔ），Ｙｄ（ｔ））の時系列変化を考慮して算出される値である。Ｘｄ’（ｔ）、Ｙｄ’（ｔ）の具体例としては、フーリエ変換によりＸｄ（ｔ）、Ｙｄ（ｔ）の低周波成分を取り出したもの、移動平均をとったもの、カルマンフィルタを通したものなどがある。低周波成分を取り出すことにより、高周波成分を除去され、見る者に対し不快感を与えるフレームの小刻みな振動を効果的に抑制することが可能となる。また、このような処理を行うことにより、反応遅れが生じてしまう場合があるが、カルマンフィルタによれば反応遅れを抑制することも可能となる。 In addition, the cutout unit 6 may obtain a frame center vector based on Equation 2.

In Equation 2, Xd ′ (t) and Yd ′ (t) are values calculated in consideration of the time series change of the head center vector (Xd (t), Yd (t)). Specific examples of Xd ′ (t) and Yd ′ (t) include those obtained by extracting low-frequency components of Xd (t) and Yd (t) by Fourier transform, those obtained by moving average, those obtained through a Kalman filter and so on. By extracting the low-frequency component, the high-frequency component is removed, and it is possible to effectively suppress the minute vibrations of the frame that cause discomfort to the viewer. In addition, there is a case where reaction delay occurs due to such processing, but the Kalman filter can suppress the reaction delay.

  The moving image editing apparatus 1 may be mounted on an imaging apparatus having an acceleration sensor. In this case, a moving image in which camera shake is effectively suppressed is picked up by the acceleration sensor, and the moving image editing apparatus 1 can further perform framing on the moving image. For this reason, it is possible to generate a moving image in which blurring in the X-axis direction is more effectively suppressed.

It is a figure which shows the example of a functional block of a moving image editing apparatus. It is a figure which shows the example of the position of the frame which a cutout part cuts out. It is a flowchart which shows the operation example of a moving image editing apparatus. It is a figure which shows the example of the frame cut out by a prior art, and the moving image after an edit. It is a figure which shows the example of the frame which the cutout part cut out, and the moving image after editing.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Moving image editing apparatus 2 Image input part 3 Temporary memory | storage part 4 Head detection part 5 Head tracking part 6 Clipping part 7 Moving image storage part

Claims (8)

Position acquisition means for acquiring the position of the object in each frame of the input moving image;
An image surrounded by the cutout frame by determining the position of the cutout frame in the frame based on the reference position preset in the frame and the position of the target object so that the target is contained in the cutout frame. Cutting out means for reconstructing a moving image by cutting out
The cutout unit obtains a difference between the reference position and the position of the object, multiplies the horizontal component of the difference by a first value, and the first value for the vertical component of the difference. A moving image editing apparatus that multiplies a smaller second value and determines the position of the clipping frame from both of the obtained values and the reference position . Position acquisition means for acquiring the position of the object in each frame of the input moving image;
An image surrounded by the cutout frame by determining the position of the cutout frame in the frame based on the reference position preset in the frame and the position of the target object so that the target is contained in the cutout frame. Cutting out means for reconstructing a moving image by cutting out
The cutout means obtains a difference between the reference position and the position of the object in a plurality of frames, and takes a time-series horizontal component in consideration of a time-series change between frames for the horizontal component and the vertical component of the difference. And a time-series vertical component, the time-series horizontal component is multiplied by a first value, the time-series vertical component is multiplied by a second value smaller than the first value, and both obtained values And a moving image editing apparatus for determining a position of the cutout frame from the reference position. The moving image editing apparatus according to claim 2 , wherein the time-series horizontal component and the time-series vertical component are low-frequency components of a time-series change of the horizontal component and the vertical component of the difference . The object is video editing apparatus according to claim 1 which is human head or face 3. A position acquisition step in which the information processing apparatus acquires the position of the object in each frame of the input moving image;
The information processing apparatus determines the position of the cutout frame in the frame based on a reference position preset in the frame and the position of the target object so that the target object is placed in the cutout frame. A clipping step for reconstructing a moving image by clipping an image surrounded by a clipping frame;
Including
In the clipping step, the information processing apparatus obtains a difference between the reference position and the position of the target object, multiplies the horizontal component of the difference by a first value, and calculates the vertical component of the difference. A moving image editing method, wherein a second value smaller than the first value is multiplied, and a position of a clipping frame is determined from both obtained values and the reference position .   A position acquisition step in which the information processing apparatus acquires the position of the object in each frame of the input moving image;
  The information processing apparatus determines the position of the cutout frame in the frame based on a reference position preset in the frame and the position of the target object so that the target object is placed in the cutout frame. A clipping step for reconstructing a moving image by clipping an image surrounded by a clipping frame;
Including
  In the clipping step, the information processing apparatus obtains a difference between the reference position and the position of the object in a plurality of frames, and performs a time-series change between frames for the horizontal component and the vertical component of the difference. Calculate a time-series horizontal component and a time-series vertical component that are considered, multiply the time-series horizontal component by a first value, and multiply the time-series vertical component by a second value smaller than the first value. A moving image editing method for determining a position of a clipping frame from the obtained both values and the reference position. For information processing equipment
A position acquisition step of acquiring the position of the object in each frame of the input moving image;
An image surrounded by the cutout frame by determining the position of the cutout frame in the frame based on the reference position preset in the frame and the position of the target object so that the target is contained in the cutout frame. A clipping step for reconstructing a moving image by clipping
A program for executing
In the clipping step, the difference between the reference position and the position of the object is obtained, the horizontal value component of the difference is multiplied by a first value, and the vertical value component of the difference is the first value. A program for causing the information processing apparatus to execute a process of determining the position of the clipping frame from both of the obtained values and the reference position .   For information processing equipment
  A position acquisition step of acquiring the position of the object in each frame of the input moving image;
  An image surrounded by the cutout frame by determining the position of the cutout frame in the frame based on the reference position preset in the frame and the position of the target object so that the target is contained in the cutout frame. A clipping step for reconstructing a moving image by clipping
A program for executing
  In the clipping step, a difference between the reference position and the position of the object in a plurality of frames is obtained, and a time-series horizontal component considering a time-series change between frames for the horizontal component and the vertical component of the difference And a time-series vertical component, the time-series horizontal component is multiplied by a first value, the time-series vertical component is multiplied by a second value smaller than the first value, and both obtained values And a program for causing the information processing apparatus to execute processing for determining the position of the clipping frame from the reference position.

Images