JP3404803B2 - Moving image recording medium, still image extracting device, moving image recording device, and still image automatic extracting method - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method and apparatus for automatically extracting a representative image from a moving image captured by a video camera or the like.

[0002]

2. Description of the Related Art As a conventional technique, for example, in a VTR, information for managing a recorded moving image is recorded on a video tape together with the moving image. As an example, VISS (VHS Index Search S
system) will be described. What is VISS?
It was developed for high-speed cueing in a VTR of the system. The video tape has a control track for recording a VISS signal for performing this high-speed cueing, in addition to a video track for recording normal image information. This VISS signal is automatically recorded on the control track when the image information is started to be recorded on the video tape. In addition, for the scene that the user wants to see,
It is also possible to record the SS signal. Using the VISS signal recorded on the video tape in this way, fast-forward reproduction called intro search can be performed.
The intro search is a process in which when a VISS signal is found during fast-forwarding, the playback state is maintained for a certain time and then fast-forwarding is repeated until the end of the tape.

[0003]

However, since the VISS signal automatically recorded in the conventional VTR is recorded when the image information is started to be recorded on the video tape, the VISS signal of the intro search is changed. When the image of the attached part is extracted, the extracted image does not sufficiently express the content of the moving image. In addition, if a VISS signal is to be added to a portion that sufficiently expresses the content of a moving image, the user has to do it manually, which takes a great deal of time and effort.

In view of the above points, an object of the present invention is to provide an apparatus and a moving image recording medium for automatically extracting a representative image that sufficiently expresses the contents of the moving image.

[0005]

In order to solve the above problems, a moving image recording medium of the present invention stores a camera operation information storage unit for storing camera operation information of a camera operated by a photographer at the time of shooting a moving image. And an image processing information storage unit that stores image processing information obtained by processing a captured image, and a shooting state information storage unit that stores shooting state information during shooting obtained by processing a signal from a sensor At least one of.

Further, the still image extracting apparatus of the present invention includes a camera operation information acquisition means for taking in camera operation information of a camera operated by a photographer when capturing a moving image, and an image obtained by processing the captured image. Image processing information acquisition means for capturing processing information, shooting state information acquisition means for capturing shooting state information during shooting obtained by processing a signal from a sensor, camera operation information from the camera operation information acquisition means, and Based on at least one of the image processing information from the image processing information acquisition unit and the shooting state information from the shooting state information acquisition unit, from the time when the photographer performs the shooting start operation to the shooting end operation Still image extraction means is provided for extracting at least one still image from the moving images captured in step 1.

Further, the moving image recording apparatus of the present invention records a moving image captured by the image pickup apparatus on a moving image recording medium, and also records still image information recording means for recording information of the still image extracted by the still image extracting means. Equipped with.

[0008]

With the above structure, the camera operation information such as the zoom of the photographer at the time of photographing, the reliability of the focus and the iris obtained by performing the image processing, the position of the subject, the presence of obstacles, etc. An evaluation is performed based on image processing information and shooting state information such as pan obtained from a sensor, and an image having a high evaluation value or satisfying a certain condition is extracted from the moving image. As a result, the extracted still image sufficiently expresses the content of the moving image.

Further, by recording the information of the extracted still image together with the moving image, it becomes possible to easily retrieve or output the representative image based on the still image information.

[0010]

FIG. 1 shows an embodiment of the moving image recording medium of the present invention. Although FIG. 1 shows a video tape as an example of the moving image recording medium, it may be another recording medium such as a video disk or an IC memory. As shown in FIG. 1, the extracted information of the representative image is recorded on the video tape along with the video signal in units of frames corresponding to the video signal. Here, the extraction information of the representative image is camera operation information, image processing information, and shooting state information. The camera operation is a recording start operation or a zoom operation when shooting with a video camera, and the recording start operation information is information indicating a recording start point by setting a flag when the recording start operation is performed. The information is information indicating the zoom magnification. Both the recording start operation information and the zoom operation information are information that can be detected based on the button operation of the video camera. The image processing information is information that is automatically or based on a video signal picked up by an image sensor and extracted by a human being, for example, the frequency and magnitude of the high-frequency component of the video signal obtained for performing focus control, Alternatively, an inter-frame difference value obtained by obtaining a difference between a luminance signal and a color signal between frames, or information obtained by extracting information such as a position and a size of a subject region from a video signal, a state of backlight or over-forward light, and further γ correction Such as value and color temperature. The shooting state information is the information obtained by detecting the shooting state of the camera with a sensor, for example, camera movement information such as panning detected by an angular velocity sensor, or the subject light amount by the light amount sensor or the aperture opening of the lens detected by the aperture opening sensor, Alternatively, there is a focus distance by detecting the lens position.

By providing the moving image recording medium with the representative image extraction information as described above, it becomes possible to extract the representative image in the moving image as described in the following embodiments.

Next, a first embodiment of the still image extracting apparatus of the present invention will be described. FIG. 2 shows a block diagram of this embodiment. In this embodiment, information for extracting a representative image together with a video signal is recorded in a moving image recording medium, the extracted information of the representative image is read out from the moving image recording medium and evaluated, and one cut is made based on the evaluation result. A typical image is extracted as a still image from (a group of moving images continuously shot between the recording start operation and the recording end operation in the camera). In FIG. 2, 1 is a reproduction signal input unit, 2 is a camera operation information acquisition unit, 3 is an image processing information acquisition unit, 4 is a shooting state information acquisition unit, 5 is a video signal acquisition unit, 6 is a representative image extraction information evaluation unit, Reference numeral 7 is a representative image management unit, 8 is a representative image storage unit, and 9 is an output device. The operation of each unit in the above configuration will be described in detail below.

First, the reproduction signal input section 1 reproduces and inputs information recorded on a moving image recording medium. The camera operation information acquisition unit 2, the image processing information acquisition unit 3, and the shooting state information acquisition unit 4 read camera operation information, image processing information, and shooting state information from the reproduction signal input to the reproduction signal input unit 1, respectively. In this embodiment, each coded information is decoded. The representative image extraction information evaluation unit 6 detects the recording start operation information included in the camera operation information, and for each image in the same cut until the next recording start operation information is detected, the representative image extraction information evaluation unit 6 Based on the information other than the recording start operation information, the image processing information, and the shooting state information, it is evaluated whether each image is an appropriate image as a representative image of cuts. Regarding the image evaluated as suitable as the representative image, the representative image management unit 7 takes in one frame image from the video signal acquisition unit 5 and stores it in the representative image storage unit 8. The output device 9 extracts and outputs the representative image stored in the representative image storage unit 8, and is a display, a printer, or the like. The operation of the representative image extraction information evaluation unit 6 will be described in detail in an example of a still image automatic extraction method described later.

In the above embodiment, information for extracting the representative image is recorded in advance in the moving image recording medium together with the video signal, and the extraction information of the representative image is read from the moving image recording medium to extract the representative image. The case was explained. However, even if a part or all of the information for extracting the representative image does not exist in the moving image recording medium, it is possible to extract the representative image by processing the video signal recorded in the moving image recording medium. Get information,
A representative image can be extracted based on the acquired information. This will be described in detail in the still image extracting apparatus of the second embodiment below.

As a comparative example, FIG . 3 shows only a video signal.
The structure of the apparatus which acquires all the typical image extraction information is shown.
In FIG. 3, 10 is an inter-frame difference value detection unit, 11 is a memory,
12 is a change amount detection unit, 13 is a cut change detection unit, 1
4 is a camerawork detection unit, 15 is a motion vector detection unit,
Reference numeral 16 is a camera work parameter estimation unit, 17 is a subject information detection unit, 18 is a moving region detection unit, 19 is a region feature amount extraction unit, 20 is a focus information detection unit, 21 is a high-pass filter, and 22 is an average value calculation. It is a department. The operation of each unit in the above configuration will be described in detail below.

First, the operations of the inter-frame difference value detecting section 10 and the cut change detecting section 13 will be described. The interframe difference value detection unit 10 includes a memory 11 for delaying the moving image signal by one frame, and a change amount detection unit 12 that obtains a difference between the moving image signals between consecutive frames.
A signal for obtaining a difference between consecutive frames of a moving image uses a luminance value, an rgb value, or the like, and the change amount detection unit 12 performs a difference calculation of image signals between consecutive frames in pixel units to obtain a difference value for each pixel. The sum is calculated and output as the interframe difference value. The cut change detection unit 13 performs threshold processing on the interframe difference value obtained by the interframe difference value detection unit 10. That is, a predetermined threshold value is compared with an inter-frame difference value, and if the inter-frame difference value is larger than the threshold value, it is considered that the image content is greatly changed between the two frames, and the cut change is performed at that portion. Judge that there was. In a video camera, a cut change occurs when a recording start operation is performed, and conversely, the recording start operation can be estimated by detecting the cut change from the image signal. Therefore, the cut change detection unit 13 outputs the recording start operation information when the inter-frame difference value exceeding the threshold value is detected. In addition,
The configuration of the inter-frame difference value detection unit 10 shown in FIG. 3 is an example, and another configuration as shown in FIG. 4 may be used. 4 in FIG.
Reference numeral 4 is a color histogram detection unit that obtains a color histogram in one frame of a moving image signal, 45 is a histogram memory that stores the obtained histogram, and 46 is a histogram difference detection unit that detects a difference in color histogram between consecutive frames. In the configuration shown in FIG. 4, the comparison is not performed for each pixel between frames but for the entire frame. However, the screen is divided into a plurality of blocks and the difference between the frames is obtained in block units. Good.

Next, the camera work detector 14 will be described. First, the operation of the motion vector detection unit 15 will be described. FIG. 5 is a diagram for explaining the position of the detected motion vector on the screen. Figure 5 (a) shows
FIG. 7 is a diagram in which M and N straight lines are arranged in a grid pattern in the vertical direction, and the MN intersections indicate the positions of motion vectors to be detected. Hereinafter, the MN intersections are referred to as grid points, and the i and jth grid points in the horizontal and vertical directions are referred to as grid points (i, j) (1≤i≤M, 1≤j≤N). .

In this comparative example, the motion vector at the grid point position is obtained by selecting a plurality of representative points around each grid point and performing representative point matching. FIG. 5 (b) is an enlarged view of the vicinity of the grid point (i, j) of FIG. 5 (a), where (2 · m + 1) · (2 · n + 1) grid points and their surroundings. The positional relationship of the representative points of is shown. Hereinafter, among the representative points of the grid point (i, j), the k-th and the l-th ones in the horizontal and vertical directions are respectively represented by the representative point (i, j, k,
l) (-m≤k≤m, -n≤l≤n). As can be seen from FIG. 5B, the representative point (i, j, 0, 0) is the grid point (i, j, 0, 0).
equal to j).

A method of obtaining a motion vector will be described below with reference to FIG. 6 showing a concrete block diagram of the motion vector detecting section 15.

It is assumed that the input of the motion vector detecting section 15 is a video signal and is set to be input once in r frames (r: a predetermined number). Here, the image at a certain time t will be referred to as the 0th frame image, and hereinafter, the image at the time (t + τ) will be referred to as the (30 · τ) th frame image.

Now, it is assumed that the image of the Rth frame has been input. The input image is first passed through a bandpass filter in the BPF 23. Here, the value of the image after the BPF processing at the coordinate position (x, y) is set to I (x, y).

On the other hand, the representative point value storage unit 24 stores the value of the representative point of the image before the r frame, that is, after the BPF processing of the (R−r) th frame. That is, the value Y (i, j, k, l) of the representative point (i, j, k, l) = I (pos_x (i, k), pos_y (j, l)) 1 ≦ i ≦ M, 1 ≦ j ≤ N, -m ≤ k ≤ m, -n ≤ l ≤ n pos_x (i, k): x coordinate of representative point (i, j, k, l) pos_y (i, k): representative point (i, The y coordinate of j, k, l) is stored.

The matching unit 25 uses the BPF 23 to the BP.
The image I (x, y) after F processing is input with the value Y (i, j, k, l) of the representative point before r frames from the representative point value storage unit 24, and at each lattice point by representative point matching. Find the motion vector of. That is, regarding the grid point (i, j),

[0024]

[Equation 1]

By searching for g and h for which the minimum is within the range of (2.G). (2.H) (-G≤g≤G, -H≤h≤H), the motion vector (g , h) is obtained.

The contents of the representative point value storage unit 24 are updated after the processing of the matching unit 25 is completed. In particular,
Coordinates pos_x (i, j, k, l) and pos_y (i, j, k, l) of the representative points stored in the representative point position storage unit 26 1 ≦ i ≦ M, 1 ≦ j ≦ N, -m The value at the representative point of the image after the BPF processing of the Rth frame is recorded by using ≤k≤m and -n≤l≤n.

As described above, the motion vector can be obtained from the two images, the input image and the image before r frames.

Next, a method of estimating the camerawork parameter from the motion vector in the camerawork parameter estimation unit 16 will be described.

The camerawork that can be estimated from the moving image is the change in the horizontal and vertical directions of the camera (panning, tilting), the change in the angle of view of the camera (zooming), the change in the horizontal / vertical / forward and backward positions of the camera (tracking, Booming, dolling) etc. are considered. In this embodiment, for simplicity, panning, tilting, and zooming are performed.
A method for estimating the type of operation will be described.

First, consider how the points projected on the image pickup surface of the camera move by the above three types of camera work. FIG. 7 is a diagram showing the positional relationship between the imaging surface of the camera and the subject, and the three-dimensional coordinates in the space of the camera are (x, y, z).
, And the two-dimensional coordinates on the imaging surface are represented by (X, Y). Further, the position of the camera is the origin of the three-dimensional coordinates, and the optical axis of the camera is the z axis. The imaging plane is located at z = F (F: focal length), and the coordinates u1 = (x1, y) of an arbitrary point on the subject.
1, z1) is projected onto U1 = (X1, Y1) on the imaging surface. Here, the relationship between the coordinates of the subject and the coordinates on the imaging surface is

[0031]

[Equation 2]

Can be expressed as First, using the coordinates in FIG. 7, consider the movement of the coordinates of the subject on the imaging surface by zooming. FIG. 8 (a) shows zooming caused by a change in focal length. When the focal length changes from F to F'as shown in the figure, the projection of the subject of u1 is U1 = (X
Move from 1, Y1) to U2 = (X2, Y2).

However, from (Equation 2), U2 is U2 = U1 · F '/ F = f · U1 However, f = F '/ F is satisfied.

Similarly, using FIG. 8B, panning,
Consider the case of tilting. Panning and tilting are equivalent to rotating the camera about the y-axis and the x-axis, respectively. When the camera rotates about the x-axis by θx as shown in the figure, the coordinate u1 in the space of the subject moves to u3. However, u3 satisfies (Equation 3).

[0035]

[Equation 3]

Assuming that the rotation angle θx with respect to x is sufficiently small, X3 = X1 and Y3 = Y1 + Fθx from the relationship of (Equation 2) with respect to the coordinate U3 = (X3, Y3) on the imaging plane after movement. Relationship is guided. Generalizing this, in the case of a camera operation in which both the x-axis and the y-axis rotate, the relationship before and after the operation of arbitrary coordinates is U3 = U1 + P, where P = (px, py) px, py: the x-axis, It can be expressed as a rotation component of the y-axis.

From the above, zooming, panning,
For general camera operation that combines tilting,
Coordinates before and after camera operation U1 = (X1, Y1), U '=
It can be seen that (X ′, Y ′) satisfies U ′ = f · U + P. Hereinafter, f is called a zoom element and P is called a rotation vector.

Therefore, it is understood that the operation amount of the camera can be estimated by obtaining the zoom element and the rotation vector.

A method of estimating the zoom element and the rotation vector from the motion vector obtained by the motion vector detecting section 15 will be described below. Here, with respect to the lattice point (i, j), the position (two-dimensional coordinate) is Ui, j, and the motion vector obtained by the motion vector detection unit 15 is vi, j.

Now, when the camera operation of the zoom element f and the rotation vector P occurs, the grid point (i, j) moves to the position of U'i, j (f, P) = f.Ui, j + P. Should be. Therefore, in order to estimate f and P of the actual camera operation, the error E (f, P) = Σ (U'i, j) with the actually moved position U ^real i, j = U i, j + vi, j It suffices to find f and P such that (f, P) -U ^real i, j) ² is minimized. Since the error E is a quadratic equation with respect to f and P, f, which minimizes the error E,
P is unique

[0041]

[Equation 4]

Is determined. However, <•, •> indicates the inner product. Therefore, the camerawork parameter estimation unit 16 inputs the motion vector vi, j and the grid point position Ui, j from the motion vector detection unit 15 and calculates f, P by (Equation 4) to perform zooming, panning, Each camerawork parameter of tilting can be estimated.

Next, the operation of the subject information detecting section 17 will be described. The subject information detection unit 17 extracts subject information such as the position, size, and color of the subject while the subject is being tracked by the video camera.
That is, when the camerawork detection unit 14 detects panning and the moving region detection unit 18 can detect a moving region, the in-region characteristic amount extraction unit 19 extracts the characteristic amount in the region from the moving region. . The operation of the moving area detector 18 will be described in more detail.

The motion area detecting unit 18 detects the motion vectors vi, j of the M / N lattice points in the screen detected by the motion vector detecting unit 15 and the motion of the camera by the panning detected by the camera work detecting unit 14. The vector Vp is input.
The moving area detection unit 18 extracts grid points that satisfy (Equation 5), and performs panning based on the connection relationship of the extracted grid points.

[0045]

[Equation 5]

A region different from the camera motion vector by is extracted. In the in-region feature amount extraction unit 19, as the feature amount in the region from the moving region detected by the moving region detection unit 18,
The center of gravity position, area, and color are extracted.

Next, the operation of the focus information detecting section 20 will be described. The focus information detector 20 is for detecting the out-of-focus state of the image, and is based on the amount of high frequency components of the image. That is, the value of the high frequency component of the video signal becomes small when the image is blurred due to defocusing of the lens. Therefore, the high-pass filter 21 takes out the high-frequency component of the image, and the average value calculation unit 22 obtains the average value of the high-frequency component in the entire screen or in the designated area.

Information for extracting the representative image can be obtained by processing the video signal as described above. In the present embodiment, the γ correction value, color temperature, backlight or over-illumination state, subject light amount, etc. were not described, but these information can also be obtained by processing the video signal. The structure and method for acquiring the information for extracting the representative image and then extracting the representative image based on the acquired information are the same as those in the first embodiment of the still image extracting apparatus, and the description thereof will be omitted.

The apparatus for acquiring the representative image extraction information described above represents the representative image based on the video signal read from the moving image recording medium when the information for extracting the representative image does not exist in the moving image recording medium. Although the image extraction information is obtained, similarly, the representative image extraction information can be obtained based on the video signal taken in from the image pickup device during photographing by the video camera. Although the configuration in this case is similar to that of FIG. 3 and the description thereof is omitted, it is not necessary to acquire the representative image extraction information from the video signal when the representative image extraction information that can be detected by the sensor included in the video camera exists. Needless to say. Further, the representative image extraction information acquired during shooting with the video camera may be recorded in the moving image recording medium together with the video signal.

Next, FIG. 9 shows the configuration of an embodiment of the moving image recording apparatus of the present invention. In this embodiment, the video camera is provided with a still image extracting device, and a still image to be extracted as a representative image is determined during shooting by the video camera, a video signal is recorded on a moving image recording medium, and the still image is extracted. The information of is recorded. In FIG. 9, the moving image recording apparatus 27 of the present invention includes a camera operation information acquisition unit 28, an image processing information acquisition unit 29, a shooting state information acquisition unit 30, and a video signal acquisition unit 3.
1, a representative image extraction information evaluation unit 32, a still image information recording unit 33, and a video signal recording unit 34. The operation of each unit in the above configuration will be described in detail below.

The camera operation information acquisition section 28 is a section for acquiring information such as a recording start operation and a zoom operation at the time of shooting with a video camera. The recording start operation information is information indicating a recording start point by setting a flag when the recording start operation is performed, and the zoom operation information is information indicating a zoom magnification. Both the recording start operation information and the zoom operation information are detected based on the button operation of the video camera. The image processing information acquisition unit 29 is a part that acquires information obtained by processing a video signal captured by an image sensor, and is, for example, the frequency or magnitude of the high frequency component of the video signal obtained for performing focus control, or the brightness between frames. Extraction of inter-frame difference values obtained from signal and color signal differences, information such as the position and size of the subject area obtained from video signals, backlit and over-light conditions, and γ correction values and color temperature To do. The shooting state information acquisition unit 30 is a part that acquires information obtained by detecting the shooting state of the camera with a sensor. For example, camera movement information such as panning detected by an angular velocity sensor, or a subject light amount or aperture opening sensor by a light amount sensor. The detected aperture of the lens or the focus distance obtained by detecting the lens position is acquired.

The representative image extraction information evaluation unit 32 detects the recording start operation information included in the camera operation information, and the camera operation information for each image in the same cut until the next recording start operation information is detected. Based on the information other than the recording start operation information, the image processing information, and the shooting state information, it is evaluated whether each image is an appropriate image as a representative image of cuts. Regarding the image evaluated as suitable as the representative image, still image information is recorded on the still image information recording medium of the recording medium 35 via the still image information recording unit 33.
Regarding the operation of the representative image extraction information evaluation unit 32,
This will be described in detail in an embodiment of a still image automatic extraction method described later.

The still image information recorded by the still image information recording section 33 will be further described below. The still image information is the still image itself evaluated as suitable as a representative image by the representative image extraction information evaluation unit 32, image processing such as reduction, or storage on a moving image recording medium corresponding to the still image. It is position information or a flag added to a moving image corresponding to a still image.

When the still image information is the still image itself or a reduced image, what is the recording position on the moving image recording medium for recording the moving image captured by the camera by the image signal acquisition unit 31 and the image signal recording unit 34? An image that is still image information is recorded at a different position or on a recording medium different from the moving image recording medium that records a moving image. For example, when the moving image recording medium is a video tape, only the images of the still image information are collectively recorded at the beginning portion of the tape or the end portion of the tape, or I provided separately from the tape.
Only still image information is recorded in the C memory.

When the still image information is the storage position information on the moving image recording medium corresponding to the still image, the video signal acquisition unit 3
1 and still image information on a recording medium different from the recording position on the moving image recording medium for recording the moving image captured by the camera by the video signal recording unit 34 or a recording medium different from the moving image recording medium for recording the moving image. The storage position information on the moving image recording medium corresponding to the still image is recorded.

When the still image information is a flag added to the moving image corresponding to the still image, the recording on the moving image recording medium for recording the moving image captured by the camera by the video signal acquisition unit 31 and the video signal recording unit 34. The still image information is recorded at the same position as the position. That is, for example, the flag of the still image information is recorded at the beginning of the video signal recorded in 1-frame units.

As described above, the video signal captured by the video camera is recorded in the moving image recording medium, the representative image is extracted from the captured moving image, and the still image information of the representative image is recorded in the recording medium. . This makes it possible to read the recorded still image information and output the representative image to a display or a printer.

Next, an embodiment of the still image automatic extraction method according to the present invention will be described. The still image automatic extraction method is a processing method in the representative image extraction information evaluation unit 6 of FIG. 2 and the representative image extraction information evaluation unit 32 of FIG.

The still image automatic extraction method according to the present invention automatically selects a typical image as a still image from the moving images continuously shot between the recording start operation and the recording end operation. To be extracted. Here, the representative image is an image selected by evaluation based on the intention of the photographer, the state of the image taken, and the state of the subject.

The photographer's intention is reflected in camera work such as zooming and panning. That is, when zooming in, the subject of interest exists on the screen, and is considered to be an important image. Also, when panning, you are moving from one scene to another,
Not considered important. Furthermore, even if you are panning, if there is an object you are tracking,
Considered to be important. In this way, it is desirable to estimate the intention of the photographer from the camera work and extract the important part as the representative image.

The state of the photographed image is a blurred image state when focus control is not properly performed at the time of photographing, a state of over-forward light or back light when iris control is inadequate, and gamma correction. A state such as when the value is inappropriate, or a state in which the aperture or focus is being adjusted. These image states can be determined based on information of focus control and iris control at the time of shooting with a video camera, or a γ correction value. Even if there is no information on focus control, iris control, or γ correction, it can be obtained by processing the video signal.
It is desirable to evaluate these image states and extract a representative image having a good image state.

The state of the subject includes the position and size of the subject being photographed, the presence of obstacles such as when a person crosses the front of the camera during photographing, and the flash flashes during photographing. A case or the like, a state in which a subject is illuminated by a spotlight, or the like. Regarding the position and size of the subject, it is desirable that the position is large in the center of the camera and the area is large, and it is desirable that there is no obstacle. Further, it is desirable not to extract the image when the flash is emitted as the representative image. Further, the image illuminated by the spotlight is the image of interest, and it is desirable to extract it as a representative image. Here, the method of detecting the position and area of the subject has been described in the subject information detection unit 17 of FIG. 3 in the third embodiment. The method of detecting a flash or an obstacle can be detected based on the interframe difference value obtained by the interframe difference value detection unit 10 in FIG. That is, in the flash, the difference value between frames suddenly changes,
It can be detected based on a predetermined threshold value. For obstacles,
Since the inter-frame difference value changes when an obstacle enters the screen and when an obstacle comes out of the screen, the inter-frame difference value exceeds the predetermined threshold value, and then the inter-frame difference value is restored within the predetermined time. If the difference value exceeds the threshold value, it can be detected that an obstacle is present in the screen. The irradiation of the spotlight can be detected based on the light amount of the subject.

Based on the knowledge for extracting the representative image as described above, a specific method of extracting the representative image will be described below. A configuration example of this embodiment is shown in FIG. In FIG. 10, 36 is a weighted addition section, 37 is a gate signal generation section, 38
Is a gate unit, and 39 is a maximum value detection unit. The zoom magnification and the subject information are input to the weighting addition unit 36, and the respective signals are weighted and added. Here, the subject information is information obtained when the subject is being tracked by the camera, and is based on the position and size of the subject at the time of tracking. The subject information has a larger value as the subject position is closer to the center of the camera and as the subject area is larger. The gate unit 38 turns on and off the switch based on the gate signal of the gate signal generation unit 37. The maximum value detecting unit 39 detects the maximum value of the values input from the gate unit 38.

The gate signal generator 37 generates a gate signal based on the high frequency component value of the pan signal and the video signal, the interframe difference value, and the like. A method of generating the gate signal is shown in FIG. FIG. 11A shows a pan signal, which is 0 during panning.
Thus, the signal is 1 when not panning. (B)
Is the high frequency component value of the video signal, and the smaller the value, the more blurred the image. (C) is a signal obtained by thresholding the signal of (b) and binarizing it. When the signal is below the threshold, it is set to zero. . (D) is a difference value between frames. (E) thresholds the signal of (d),
A process of setting 0 when the value is equal to or more than the threshold value, and also setting a section between 0 and 0 when the signal of (d) exceeds the threshold value again within a predetermined time after the threshold value is exceeded. I do. That is, if the inter-frame difference value increases by itself, it is determined that an image abnormality due to a flash or the like has occurred,
It is set to 0 only when the interframe difference value is equal to or greater than the threshold value. However, as described above, when an obstacle passes in front of the camera, the inter-frame difference value takes a plurality of peaks. Even if it is less than or equal to 0, it is set to 0. A gate signal is generated by taking the logical product of the three signals (a), (c), and (e) that are binarized as described above.

As described above, by obtaining the image having the maximum evaluation value in the configuration shown in FIG. 10, the panning period, the image blurring period, and the flash or obstacle existence period are determined. An image in which the zoom magnification is high and the subject is largely reflected in the center of the screen can be extracted as a representative image from the removed images. The maximum value detection unit 3
The maximum value is detected in 9 even if the whole one cut
Alternatively, it may be a plurality of sections within one cut.

In FIG. 10, one evaluation value is obtained from a plurality of inputs by the configuration of the weighted addition unit 36, the gate signal generation unit 37, and the gate unit 38, but the present invention is not limited to this configuration, and fuzzy inference is performed. It is possible to use a rule based on such rules, or a configuration obtained by a neural network. Further, although the processing of the information regarding the γ correction value, the light amount of the subject, the backlight or over-forwarding condition, the aperture opening, and the focus distance is not shown in the configuration of the present embodiment, these signals can be used in the same manner. it can. That is, γ
A gate signal may be generated so as not to be extracted as a representative image when the correction value, the aperture value, or the focus distance value is fluctuating, or when the backlight or over-illumination state is present. Alternatively, the evaluation value may be increased by detecting that the spotlight is emitted from the amount of light of the subject.

The configuration of the embodiment of the still image automatic extraction method described above evaluates all moving images taken between the time when the photographer performs the photographing start operation and the photographing end operation. Although the representative image is extracted by the photographer, the image may be evaluated after a lapse of a predetermined time after the photographer starts the photographing operation, and the image at the time when the predetermined condition is satisfied may be extracted as the representative image. An example of this configuration will be described below.

The structure of this embodiment is shown in FIG. In FIG. 12, 40 is a timer, 41 is a gate unit, and 42 is an evaluation unit. The timer 40 measures the elapsed time after the shooting start operation is performed, and generates a gate signal to open the gate of the gate unit 41 when a certain time has elapsed from the start of shooting. The evaluation unit 42 evaluates whether the high-frequency component value of the focus that has passed through the gate unit 41 and the inter-frame difference value satisfy the conditions. The condition in the evaluation unit 42 is that the high frequency component value of focus is equal to or higher than a predetermined threshold value and the inter-frame difference value is equal to or lower than the predetermined threshold value. When the condition is satisfied, the evaluation unit 42 stops the subsequent evaluation and extracts a still image at the time when the condition is satisfied as a representative image. In this embodiment, the signals used for evaluation are only the high-frequency component value of focus and the inter-frame difference value, but other signals such as pan and zoom may be used.

Further, the configuration of another embodiment of the still image automatic extraction method will be described. The structure of this embodiment is shown in FIG. 13, and the same parts as those in FIG.
In the present embodiment, evaluation is performed from the number of images obtained by multiplying the number of images captured between the image capturing start operation and the image capturing end operation by a fixed ratio, and the image at the time when a predetermined condition is satisfied is extracted as a representative image. It is a thing. In the configuration of FIG. 13, for images captured between the shooting start operation and the shooting end operation,
The image after the intermediate frame is evaluated. Therefore, the intermediate frame detection unit 43 detects an intermediate frame of frames captured from the shooting start operation to the shooting end operation, and when the intermediate frame is detected, the gate unit 4 is detected.
Generate a gate signal to open the gate of 1. The operations of the gate unit 41 and the evaluation unit 42 are the same as the configuration of FIG.

[0070]

[0071]

The still image extraction apparatus of the present invention] based on image extraction information, it is possible to extract a still image to fully represent the content of the moving image from the moving image as a representative image, videos The contents of the image can be grasped in a short time.

Further, the moving image recording apparatus of the present invention records the video signal photographed by the camera in the moving image recording medium, extracts the representative image from the photographed moving image, and records the still image information of the representative image. By recording on the medium,
It is possible to read the recorded still image information and output the representative image at high speed to a display or a printer.

[Brief description of drawings]

FIG. 1 is a diagram showing a moving image recording medium in an embodiment of the present invention.

FIG. 2 is a diagram showing a configuration of a still image extracting apparatus according to an embodiment of the present invention.

FIG. 3 is a diagram showing a configuration of a representative image information extraction device in a still image extraction device of a comparative example of the present invention.

FIG. 4 is a block diagram showing another configuration of an interframe difference value detection unit in the still image automatic extraction apparatus of the comparative example .

FIG. 5 is a diagram illustrating lattice points for detecting a motion vector in a motion vector detection unit in the still image automatic extraction device of the comparative example .

FIG. 6 is a diagram showing a configuration of a motion vector detection unit in the still image automatic extraction apparatus of the comparative example .

FIG. 7 is a diagram showing a positional relationship between an imaging surface of a camera and a subject.

FIG. 8 is a diagram showing a positional relationship between an image pickup surface of a camera and a subject in zooming and tilting.

FIG. 9 is a diagram showing a configuration of a moving image recording apparatus according to an embodiment of the present invention.

It illustrates a configuration of a representative image extracting information evaluation unit in Figure 10 the embodiment

FIG. 11 is a diagram for explaining the operation of the gate signal generation unit in the representative image extraction information evaluation unit .

FIG. 12 is a diagram showing a different configuration of the representative image extraction information evaluation unit.

FIG. 13 is a diagram showing a different configuration of the representative image extraction information evaluation unit.

[Explanation of symbols]

1 Playback signal input section 2, 28 Camera operation information acquisition unit 3, 29 Image processing information acquisition unit 4, 30 Shooting status information acquisition unit 5, 31 Video signal acquisition unit 6, 32 Representative image extraction information evaluation unit 7 Representative image management section 8 Representative image storage 9 Output device 33 Still image information recording section 36 Weighting adder 37 Gate signal generator 38, 41 Gate 39 Maximum value detector 40 timer 42 Evaluation Department 43 Intermediate frame detector

─────────────────────────────────────────────────── ─── Continuation of the front page (56) References JP-A-2-151188 (JP, A) JP-A-6-217254 (JP, A) (58) Fields investigated (Int.Cl. ⁷ , DB name) H04N 5/76-5/956 H04N 5/225-5/243

Claims (6)

(57) [Claims] Claim: What is claimed is: 1. A camera operation information acquisition unit for capturing camera operation information of a camera operated by a photographer when capturing a moving image, and an image processing information acquisition for capturing image processing information obtained by processing the captured image. Means, a photographing state information acquisition means for taking in photographing state information during photographing obtained by processing a signal from the sensor, and a photographer performing a photographing start operation.
From the moving image shot between the
Image extraction to extract at least one still image from
Image processing from the image processing information acquisition means
Information and the shooting state information from the shooting state information acquisition means.
At least one information and the camera operation information acquisition means
The still image extracting means based on the camera operation information
A still image extraction device , characterized in that a still image is extracted by. 2. A camera operation information acquisition means for acquiring camera operation information of a camera operated by a photographer when shooting a moving image, and an image processing information acquisition for acquiring image processing information obtained by processing the captured image. Means, a photographing state information acquisition means for taking in photographing state information during photographing obtained by processing a signal from the sensor, and a photographer performing a photographing start operation.
From the moving image shot between the
Image extraction to extract at least one still image from
Means and a still image information recording device for recording information of a still image
A moving image recording medium that includes a step and records a moving image captured by an image capturing device.
And from the image processing information acquisition means.
Image processing information and shooting condition from the shooting condition information acquisition means
Status information and at least one camera operation information
The still image based on the camera operation information from the obtaining means.
Information of the extracted still image extracted by the extraction means
Moving image recording apparatus characterized by recording a. 3. For photographer extracting at least one still image from the moving image captured until the shooting end operation after the photographing start operation, the image captured
Image processing information obtained by processing and the signal from the sensor
At least one of the shooting state information during shooting obtained by processing
Information and the camera such as the zoom of the imaging device and the shooting start operation.
La operation information as input, based on the still image extraction knowledge obtains an evaluation value for each image, a still image automatic extraction method characterized by extracting the high image evaluation value. 4. An image satisfying a certain condition in order to extract at least one still image from the moving images photographed between the photographing start operation and the photographing end operation by the photographer. On the other hand, at least one of the camera operation information of the imaging device, the image processing information obtained by processing the captured image, and the shooting state information during shooting obtained by processing the signal from the sensor is input, A still image automatic extraction method, characterized in that an image in which the camera operation information, the image processing information, and the shooting state information satisfy predetermined extraction conditions is extracted. 5. A certain period of time from the shooting start operation in order to extract at least one still image from the moving images taken between the shooting start operation and the shooting end operation by the photographer. For the image after the elapse, the camera operation information such as the zoom of the imaging device, the image processing information obtained by processing the captured image, and the shooting state information during shooting obtained by processing the signal from the sensor 5. The still image automatic extraction method according to claim 4 , wherein at least one of the inputs is extracted, and an image in which the camera operation information, the image processing information, and the shooting state information satisfy predetermined conditions is extracted. 6. A shooting start operation to a shooting end in order to extract at least one still image from the moving images taken between the shooting start operation and the shooting end operation by the photographer. An image obtained by processing camera operation information such as the zoom of the imaging device and the captured image for the image captured after the number of images obtained by multiplying the number of images captured by the operation by a certain ratio An image in which at least one of the shooting state information during shooting obtained by processing the signal from the processing information and the sensor is input, and the camera operation information, the image processing information, and the shooting state information satisfy predetermined conditions. 5. The automatic still image extracting method according to claim 4, wherein