JP4046920B2 - Moving object extraction device - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image signal processing apparatus, and more particularly to a moving object extraction apparatus that extracts a moving object from a moving image.
[0002]
[Prior art]
One method for extracting a moving part from a moving image, that is, a moving object, is a method of using a difference between an image as a background and an input moving image as a moving object.
As a specific method, there is a method of creating a background image from a moving image to be processed and taking a difference (Sakaida, Kinji, “Examination of moving object extraction method using background difference”, The University of Science, D-11-113, 1999). This is to generate a background image by statistically processing a moving image over a long time in the time direction, and using the fact that the still image portion corresponding to the background has the highest frequency in the luminance histogram in the time direction. Yes.
[0003]
However, there is no guarantee that only the mode value of the histogram will be the background depending on the fluctuation of the illumination conditions at the time of shooting. Therefore, a range that includes the mode value and is in the vicinity of the mode value is provided, and the pixels are within that range. Only when the background image. On the contrary, the moving object is a pixel within a certain range defined further outside the vicinity of the mode value. In the process so far, apart from the pixels that are the background and the moving object, the pixels that cannot be distinguished from either the background or the moving object remain. This part is classified as an undetermined area into a background pixel or a moving object in the frame image using a watershed method described later.
[0004]
Here, since the application of the watershed method is limited to the frame image, the finally extracted moving object is not considered in the temporal direction, that is, the continuity between frames, so it was viewed as a moving image. Sometimes the outline of an object appears to rattle.
In this method, it is assumed that there is no camera movement or camera movement during shooting, such as dolly, pan, or zoom, or that compensation has been made by preprocessing.
[0005]
The watershed method is a region growing method often used for an object extraction method in one frame image, that is, a still image. In particular, the watershed method with a marker is often used because it can reduce the amount of calculation compared to a simple watershed method. The literature (Naemura, Mizutani, Izumi, “Morphological Segmentation Using Prior Knowledge Information in Sports Programs”, Video Media Processing Symposium, I-3.15, 1997) proposes a marker selection method for the watershed method. Yes. This is a method in which the color of the grass on the ground in the case of a sports scene or the color of a player's uniform is manually extracted in advance as a representative region, and the marker is determined using the color range for a histogram.
[0006]
[Problems to be solved by the invention]
As described above, the uncertain area that is ambiguous about the boundary between the background area and the moving object area in the moving image belongs to any of those areas, and the background pixel or moving object using the watershed method in the frame image In determining which one of them belongs, since the continuity between frames is not taken into consideration, there is a problem to be solved that the outline of the object in the moving image may appear to be blurred.
[0007]
An object of the present invention is to prevent a moving object extraction apparatus that extracts a moving object from a moving image so that the outline of the extracted object does not appear to rattle.
[0008]
[Means for Solving the Problems]
In order to achieve the above object, the moving object extraction apparatus of the present invention uses a spatio-temporal watershed method to determine an uncertain area that is unclear as to which of the background area and the moving object area in the moving image belongs to the boundary. A region determination unit that determines to belong to either the background region or the moving object region, and the region determination unit based on the spatio-temporal watershed method is configured to detect pixels of a predetermined pixel region over a plurality of continuous frame images. Among them, the means for setting the pixels in the undetermined region of the predetermined frame image as the processing target pixels, and the edge strength including the spatio-temporal over the plurality of frame images, all the elements constituting the pixel region of the plurality of frame images The smallest edge strength among the pixels excluding the undetermined area in the pixel area extending over the plurality of frame images and the means for setting the pixel Means for extracting a pixel having a pixel , and means for determining whether the extracted pixel belongs to a background area or a moving object area. A pixel that is not included in the undetermined region among neighboring pixels that are adjacent to each other and that has the lowest edge strength, and that is set as a pixel in either the background region or the moving object region. To do.
[0009]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, the present invention will be described in detail based on an embodiment of the invention with reference to the accompanying drawings.
FIG. 1 is a block diagram showing an embodiment of the moving object extraction apparatus of the present invention.
In FIG. 1, 1 is an undetermined region extraction circuit, 2 is a first memory, 3 is a second memory, 4 is a spatiotemporal edge image extraction circuit, 5 is a third memory, 6 is a fourth memory, and 7 is a spatiotemporal. It is an image area growing circuit.
FIG. 2 shows a background area, a moving object area, and an undetermined area in the moving image.
[0010]
The operation will be described.
First, a moving image signal is supplied to the undetermined image region extraction circuit 1 and the first memory 2. In addition to the above, the undefined region extraction circuit 1 is also supplied with a background region signal and a moving object region signal as shown in the figure.
Here, the background area signal and the moving object area signal are the documents referred to in the description of the prior art (Sakaida, Kinji, “Examination of moving object extraction method using background difference”, Shingaku Sodai, D-11-113, 1999) can be used, but as another method, a method specified by a person each time may be used.
[0011]
The undetermined area extraction circuit 1 outputs an area in the moving image other than the area occupied by the background area and the moving object area as an undetermined area (undetermined area signal). The indeterminate area signal is stored in the second memory 3 for a plurality of frames (in the embodiment described below, for three frames) and becomes a plurality of frames indeterminate area signal A (see FIG. 1). It is supplied to the circuit 7.
[0012]
On the other hand, the first memory 2 stores a plurality of image signal frames (the same number as the second memory 3 (for 3 frames)) and outputs a plurality of frame image signals. This multi-frame image signal is sent to the spatio-temporal edge image extraction circuit 4 in the next stage. The spatio-temporal edge image extraction circuit extracts the spatio-temporal edge image signal B (see FIG. 1) from the multi-frame image signal and outputs it. To do. Here, the spatio-temporal edge image signal is extracted from an edge extraction method in a frame image of a two-dimensional plane, for example, a Sobel filter (for example, Aoiin, Nagao, “Image processing and recognition”, published by Shoshodo, (1992) ) (See the description on pages 35 to 36)) is also performed between a plurality of frames (in the time direction), and is added to the calculation results in space. The obtained spatiotemporal edge image signal B is supplied to the spatiotemporal image region growing circuit 7.
[0013]
The third memory 5 and the fourth memory 6 are respectively supplied with the background area signal and the moving object area signal, and the signals for the same number of frames (for 3 frames) as in the second memory 3 (respectively, FIG. 1 Are represented by symbols C and D), and both are supplied to the spatio-temporal image region growing circuit 7.
[0014]
As described above, the spatiotemporal image region growing circuit 7 is supplied with the multi-frame undetermined region signal A, spatio-temporal edge image signal B, multi-frame background region signal C, and multi-frame moving object region signal D. It will be. In the spatiotemporal image region growing circuit 7, the uncertain region (multiple frame unconfirmed region signal A) supplied to the circuit 7 is grown by the watershed method expanded to the spatiotemporal image, and the same region is the background region. Or whether it is a moving object area.
[0015]
Here, the normal intra-frame watershed method will be described.
The normal in-frame watershed method considers an image as a topographic structure, injects water from the lower elevation of the terrain, and divides the area by virtually constructing dams in watersheds that divide the basin. It is. FIG. 3 shows an example of a normal intraframe watershed method. The watershed method with marker is a method that can reduce the calculation time compared to processing all images by providing a marker at the target part and injecting water only to that part. is there. In addition, an edge image that is a brightness gradient in the space of the image is used for an image that is considered to be a topographic structure.
[0016]
On the other hand, the spatio-temporal watershed method extends the above-described watershed method and applies the watershed method to a spatiotemporal image composed of a plurality of frames. Since the spatio-temporal image is a three-dimensional image, it cannot be explained by the image of injecting water as in the normal watershed method.
[0017]
FIG. 4 shows an application example of the spatiotemporal watershed method. The example shown in FIG. 4 is for a case where an image is composed of three frames, the previous frame, the frame, and the next frame.
In FIG. 4, a background area (multiple frame background area signal C), a moving object area (multiple frame moving object area signal D), an undefined area (multiple frame undefined area) on the spatio-temporal edge image signal B image. Signal A) is depicted in gray, black and white, respectively. If the region growing process by the spatio-temporal watershed method is performed in a window of 3 × 3 pixels, for example, the portion surrounded by a thick line in FIG. This is the pixel at the center of the pixel window (the pixel shown in black at the center of the portion surrounded by the thick line).
[0018]
The region growth processing is performed in order from the pixel having the smallest edge strength (edge level value) among the pixels of the undefined region (white) over three frames. This procedure is just equivalent to injecting water from the lower elevation of the terrain in the normal in-frame watershed method. Indeterminate in the temporal and spatial neighboring pixels (for example, 26 pixels in the 3 × 3 × 3 pixel space-time) of the pixel (pixels shown in black in the 3 × 3 pixel window in FIG. 4) in the frame A pixel having a minimum edge level value is searched for among pixels that are not the region (white).
[0019]
The search is performed as follows. That is, if this neighboring pixel belongs to the background area (gray), the pixel belongs to the background area, and if it belongs to the moving object area (black), the pixel belongs to the moving object. And If the pixel cannot belong to either one, a searched mark is attached to the pixel, and the search is performed again when searching for another neighboring pixel. This search is continued until all the pixels in the undetermined region (white) belong to either the background region or the moving object.
[0020]
FIG. 5 is an enlarged view of a portion (a window of 3 × 3 pixels) surrounded by a thick line in FIG. Note that the numbers (3 to 9 in this example) attached to each pixel are edge level values.
In FIG. 5, the pixel indicated by the edge level value 5 in the frame is not an undetermined region among the 26 spatio-temporal pixels of the pixel, and the pixel having the smallest edge level value is the lower left background region. Since it is a pixel (edge level value 3), it is assumed that the pixel belongs to the background area.
[0021]
When all of the pixels in the undetermined area shown in white in FIG. 4 are grown, and the pixels in the undetermined area belong to either the background area or the moving object, they are newly added by the growth of the area. An object (moving object signal) that is not visible from the spatio-temporal image region growing circuit 7 shown in FIG.
[0022]
6 shows a case where a moving object is extracted from a moving image using a normal intra-frame watershed method, and a moving object is extracted from a moving image using a spatio-temporal watershed method according to the present invention. The moving objects obtained in this case are shown in comparison.
In FIG. 6, (a), (c), and (e) are when the intraframe (spatial) watershed method is used, and (b), (d), and (f) are when the spatiotemporal watershed method is used. 4 shows moving objects extracted corresponding to the previous frame, the corresponding frame, and the next frame (shown as the 125th frame, the 126th frame, and the 127th frame, respectively) shown in FIG. It can be seen that the object extracted using the spatio-temporal watershed method has less time wobble than the object extracted using the spatial watershed method, and an accurate object is obtained.
[0023]
In the above, when moving objects are extracted from a moving image, the uncertain areas that are ambiguous about the boundary between the background area and the moving objects are grown using the spatio-temporal watershed method. It was explained that moving objects that can not be seen can be extracted, but the edge signal of any image is used, not limited to the uncertain image area that belongs to the background image area of the moving image or the boundary of the moving object. It is also possible to realize an area dividing circuit for dividing an image using a spatio-temporal watershed method.
[0024]
【The invention's effect】
As described above, according to the present invention, when a moving object is extracted from a moving image, the spatio-temporal watershed method is applied to an undetermined region in which the determination of the background region and the moving object is ambiguous. Shaking of the object outline between frames can be suppressed. As a result, moving objects can be extracted with high accuracy.
[Brief description of the drawings]
FIG. 1 is a block diagram illustrating an embodiment of a moving object extraction device of the present invention.
FIG. 2 shows a background area, a moving object area, and an undetermined area in a moving image.
FIG. 3 shows an example of a normal intra-frame watershed method.
FIG. 4 shows an application example of a spatio-temporal watershed method.
5 is an enlarged view of a portion surrounded by a thick line in FIG.
FIG. 6 shows a case where a moving object is extracted from a moving image using a normal intra-frame watershed method and a moving object is extracted from a moving image using a spatio-temporal watershed method according to the present invention. The moving objects obtained in this case are shown in comparison.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 1 Undetermined area | region extraction circuit 2 1st memory 3 2nd memory 4 Space-time edge image extraction circuit 5 3rd memory 6 4th memory 7 Space-time image area growth circuit

Claims (1)

An undetermined area that is unclear as to which area belongs to the boundary between the background area and the moving object area in the moving image is made to belong to either the background area or the moving object area by the spatio-temporal watershed method. Comprising region determining means for determining
The area determination means includes
Means for setting a pixel in an undetermined region of a predetermined frame image as a processing target pixel among pixels of a predetermined pixel region extending over a plurality of continuous frame images;
Means for setting edge strength including time and space over the plurality of frame images to all pixels constituting a pixel region of the plurality of frame images;
Means for extracting a pixel having the smallest edge strength among pixels excluding the undetermined region in the pixel region extending over the plurality of frame images;
Means for determining whether the extracted pixel belongs to a background area or a moving object area;
Based on the determination result, the processing target pixel is not included in the undetermined region among neighboring pixels adjacent to the processing target pixel in time and space, and the background region or motion region to which the pixel having the lowest edge strength belongs. A moving object extraction device comprising: a setting unit configured as one of pixels in the object region .

Images