JP2822007B2 - Extraction and tracking of contours in images - Google Patents

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a method for extracting and tracking contours in an image, and more particularly to a method for extracting and tracking contours using an active contour model.

[0002]

2. Description of the Related Art In recent years, in fields such as computer vision and image communication systems, techniques for automatically and non-contactly detecting the movement and posture of a person have been developed. And the importance of such techniques is increasing.

In such a technique, generally, the movement and posture of a person are detected based on an image obtained by imaging a person or the like. Conventional methods for detecting the motion and posture of a person from an image include the following methods.

[0004] Ohtani et al. Have proposed a method of detecting the motion and posture of a person using a silhouette image ("Study on detection of posture of a person from multiple images using a genetic algorithm", IEICE Technical Report. , PRU93-122,
pp. 47-54, (1994-1)). Further, a method of detecting a motion and a posture of a person using an edge image has also been proposed.

Conventionally, such a detection method has been proposed. However, if a complicated contour can be tracked in a moving image, information relating to the movement of an edge inside the silhouette can be obtained. It is considered that detection can be realized.

As a method of tracking the contour of an object such as a person, M. An active contour model called Snakes by Kass et al. (Active Contour M
A method of tracking contours using odels has been proposed (“Snakes: Active ContourModels”, IJCV vol. 1).
(No. 3), pp. 321-331, (1988)).

In this method, a human sets an initial position of such an active contour model (hereinafter referred to as ACM) in advance, and then performs image processing.
In the image processing, the contour is extracted by changing the ACM by a method of minimizing the energy on the image. However, in the contour tracking method, it is necessary for a person to accurately set the initial position of the ACM in order to accurately track the contour.

[0008] As a method of developing the portion relating to the setting of the initial position in the contour extraction method using the ACM using the snake, a growing snak is used.
e) and a method using it has been proposed (“Controlling Growing snakes by Using Key Po
ints ", Olof Henricsson and Walter Neuenschwander
, In Proc. ICPR, pp. 68-73, 1994). The glowing snake is also ACM.

In the contour extraction method, a person initially sets a point serving as a seed for a glowing snake, and then extends the glowing snake. That is, the method is a method of growing a snake. In the method, the contour is extracted by stopping the growing glow snake at the edge of the object obtained by the image processing. In that method, it is necessary for a human to specify a point serving as a seed for the glowing snake on the edge of the object in the image.

Next, an example in which such a conventional contour extraction method is used for setting an initial position for tracking a contour in a moving image will be described. FIG. 10 is a flowchart showing a processing procedure when a conventional contour tracking method is used for tracking a contour in a moving image.

Referring to FIG. 10, first, at step S41
Then, an image is captured. Then, in step S42, edges are extracted from the image thus captured. By extracting the edge, an edge image from which the edge of the object is extracted is obtained.

Next, in step S43, a potential image is generated based on the edge image. The potential image is an image in which an edge is expressed by the level of energy of the image, and the energy of the edge portion is lower than that of the other portions.

Then, before proceeding to the next process, in step S44, the initial position of the ACM is set by a human. That is, while a human looks at an image, an initial position of the ACM is set along an edge in the image. The setting is made by manual input.

Next, in step S45, the ACM set in step S44 is tracked on the generated potential image. The tracking is performed by performing a process of attracting the ACM to a low energy portion (edge portion) in the potential image.

The processing of steps S41 to S45 is processing for the first frame of the image of a plurality of frames. Among them, steps S41, S42, S4
The contours in the moving image are tracked by repeatedly executing the processing such as 3 and S45 for each frame of the image.

[0016]

However, in the conventional contour tracing method as described above, before performing the process of tracing the edge by the ACM, the human being accurately sets the initial ACM for the edge to be tracked. is required. For this reason, the conventional contour tracking method has a problem that it is not suitable for tracking a contour that repeats occlusion and appearance as shown in a moving image.

Furthermore, in such a conventional contour tracking method, when an edge is locally occluded in a moving image, the deformation of the ACM corresponding to the occluded portion is A
There is also a problem that the shape of the entire CM is affected.

The present invention has been made to solve such a problem, and is suitable for tracking a contour in a moving image by enabling automatic setting of an initial position of an ACM. Another object of the present invention is to provide a method for extracting and tracking contours in an image.

[0019]

SUMMARY OF THE INVENTION According to the present invention, in a plurality of frames obtained by imaging a moving object over time, an outline of the object is extracted and tracked in each frame image. A method for extracting and tracking an inside contour, comprising a first processing step and a plurality of second processing steps.

The first processing step is a step of performing processing corresponding to the first frame of the plurality of frames, and includes a first edge extraction step, a first model generation step, and a first model combination step. Including.

The first edge extraction step extracts an edge of an object in the image of the first frame. In the first model generation step, based on the extracted edge information,
Generate multiple active contour models along the contour of the object. The first model combining step combines the plurality of generated active counter models based on a predetermined combining condition.

The second processing step is a step of performing processing corresponding to each of a plurality of frames after the first frame, each of which includes a second edge extraction step,
Potential image generation step, model tracking step,
A model splitting step, a model disappearing step, a second model generating step, and a second model combining step.

The second edge extraction step extracts an edge of the object in the image of the corresponding frame. In the potential image generating step, a potential image corresponding to the extracted edge is generated based on the extracted edge information. The model tracking step causes the active contour model to be tracked to the contour of the original image based on the generated potential image and the active contour model combined in the processing corresponding to the immediately preceding frame.

In the model splitting step, in the tracked active contour model, as a result of tracking the original image, the ACM spanning two contours is split based on a predetermined splitting condition. The model extinguishing step extinguishes, based on a predetermined extinguishing condition, an active counter model in a portion where tracking of the potential image has failed.

In the second model generation step, based on the information of the edge extracted in the second edge extraction step,
Generate multiple active counter models. In the second model combining step, a new plurality of active counters along the contour of the object is obtained by combining the successfully tracked active counter models with the plurality of active counter models generated in the second model generating step. The models are determined and their active counter models are combined based on predetermined combining conditions.

[0026]

According to the present invention, the first processing step is executed corresponding to the first frame among the images of the plurality of frames, and the second processing steps are executed corresponding to each of the plurality of subsequent frames. A processing step is performed.

In the first processing step, edges of the object in the image of the first frame are extracted, and a plurality of active counter models are generated based on the information on the extracted edges. Then, by combining these active counter models, an active counter model corresponding to an edge to be tracked in a subsequent frame is initialized based on the image of the first frame.

That is, in the first processing step, the active counter model is automatically initialized based on the edge in the image of the first frame.

Then, in each of the plurality of second processing steps, a potential image is generated based on the edges extracted in the image of the corresponding frame. Then, the active counter model combined in the processing corresponding to the immediately preceding frame tracks the potential image.

The active contour model that spans the two contours of the tracked active contour model is split. Further, the portion where the tracking of the potential image has failed is deleted based on a predetermined deletion condition. Therefore, in the active frame model, for example, in the image of the corresponding frame, a portion corresponding to an unoccluded edge remains, and a portion corresponding to an occluded edge disappears.

Further, a plurality of active counter models are generated based on the edges extracted in the image of the corresponding frame. The active canter model is
It is intended to supplement the part of the disappeared active counter model. Then, the active counter model that has been successfully tracked and the active counter model generated in this way are combined, so that a plurality of active counter models in which the disappeared portions are supplemented are obtained. Further, by combining the active counter models, an active counter model corresponding to the edge to be tracked in the next frame is newly set.

By repeatedly executing such a second processing step, in a plurality of frame images,
Extraction and tracking of the contour of the object are sequentially performed in frame order.

[0033]

Next, an embodiment of the present invention will be described in detail with reference to the drawings.

First, an outline of this embodiment will be described. In the method for extracting and tracking the contour in an image shown in the following embodiment, the extending direction of the edge of the object in the image does not change suddenly, and the difference in luminance on both sides of the edge is constant. ACM is automatically applied to a certain edge portion. Such an initial setting of the ACM is performed based on information on the edge of the first frame of the plurality of image frames.

Further, in each subsequent frame,
Based on the extracted edges, the ACM is automatically fitted, and in each frame the automatically fitted ACM
The CM tracks the edge portion of the potential image obtained from the edge information in the next frame.

FIG. 1 is a block diagram showing a configuration of an automatic contour extraction / tracking device according to an embodiment of the present invention. Referring to FIG. 1, the device includes a camera 1 and a data processing device 2.

The camera 1 images a movable object 3 such as a person over time. Image information obtained by imaging with the camera 1 is provided to the data processing device 2. That is, a plurality of frames of image information (moving image information) are provided from the camera 1 to the data processing device 2 in accordance with the lapse of time of imaging.

The data processor 2 executes data processing for automatically extracting and tracking the outline of the imaged object 3 based on the given image information.

Next, the contents of the data processing in the data processing device 2 will be described in detail. FIG. 2 is a flowchart showing a processing procedure of the automatic contour extraction / tracking method according to the embodiment of the present invention. In FIG. 2, regarding the processing of the images of a plurality of frames, the processing corresponding to each frame is indicated by steps connected in the vertical direction.

First, processing corresponding to the first frame of the image of the plurality of frames is performed. In the image input step of step S11, an image from the camera 1 is input.
Next, an edge of the image is extracted in an edge extraction step of step S12. As a result, an edge image as shown in FIG. 3 is obtained. Here, referring to FIG.
The edge image shows a plurality of edge pixels 4, 4,... Along the edge portion of the object 3 (see FIG. 1).

Next, in the ACM generation step of step S13, based on the edge image obtained in step S12, the ACM of the minimum unit (hereinafter, referred to as the element of ACM)
Is generated.

Then, referring to FIG. 4, step S13
In the ACM generation step, some of the edge pixels 4, 4,... Obtained in step S12 are eliminated as shown in FIG. Thereafter, an ACM element as shown in FIG. 5 is generated.

This thinning-out is performed because the element of the ACM generated in step S13 is the AC defined as the connection condition in the next connection step (step S14).
This is to ensure robustness with respect to the condition of the inclination of the end of M.

Next, referring to FIG. 5, in edge pixels 4, 4,... After being decimated as shown in FIG.
Two adjacent edge pixels 4 and 4 are combined. By such a combination, a plurality of ACM elements 40, 40,... Are generated.

Next, in the ACM combining step of step S14, a process of combining ACM elements 40, 40,... Is performed. Hereinafter, the contents of the processing will be described in detail.

FIG. 6 is a schematic diagram showing the state of ACMs combined in the ACM combining step. In FIG. 6, AC
ACM A, which is an ACM in which a plurality of elements of M are combined,
A state in which ACM B is combined is shown.

As shown in FIG. 6, two ACM A
And B are represented by a series of point coordinates as shown in (1) and (2) below.

[0048] _{A: (ax i, ay i} ) i = 1, ..., n (1) B: (bx i, by i) i = 1, ..., m (2) 2 single ACM A being thus represented For B and B, the following three binding conditions are defined. The first joining condition is as follows. That is, two ACs
Since M A and B must be adjacent,
The distance between these ends is a constant value threshold1
The following is the first combination condition. The first coupling condition is shown in the following equation (3).

[0049]

(Equation 1)

The second joining condition is as follows. That is, since the two ACMs A and B must be smoothly connected, the second coupling condition is that the difference between the inclinations of the ends of the ACMs is equal to or smaller than a predetermined value threshold2. The second coupling condition is represented by the following equation (4).

[0051]

(Equation 2)

The third combination condition is as follows. That is, since two ACMs A and B are models representing the same edge portion, each ACM A and B
Both sides of CM (d1 and d2 or d3 and d4)
Must be similar for the two ACMs A and B.

Therefore, when the luminance difference corresponding to ACM A is C _an and the luminance difference corresponding to ACM B is C _b1 , the difference between the luminance difference C _an and the luminance difference C _b1 is a constant value. The third joining condition is that the threshold value is equal to or less than threshold3. The third coupling condition is represented by the following equation (5).

[0054]

(Equation 3)

In the ACM combining step, ACMs satisfying the above three combining conditions are combined with each other to obtain a combined ACM C. The combined ACM C
Is represented by a series of point coordinates as shown in the following (6). The sequence of coordinates of the point is obtained by combining the sequence of coordinates of the original ACM A point represented by the following equation (7) and the sequence of coordinates of the original ACM B point represented by the following equation (8). It will be.

[0056]

(Equation 4)

As described above, in the ACM combining step of step S14, the ACM element 4 satisfying the predetermined combining condition is used.
.. And the ACMs to which the ACMs are combined are appropriately combined to form ACs corresponding to the edge portions of the image.
Get M. That is, in the processing corresponding to the first frame, the initial setting of the ACM position is automatically performed.

Next, a process corresponding to the second frame is executed. The processes after the second frame have the same contents for each frame. Here, the processing corresponding to the n-th frame shown in FIG. 2 will be described as the processing corresponding to the second frame.

In the processing corresponding to the second frame, the input image step (step S21) and the edge extraction step (step S22) are executed as in the case of the first frame. Thereby, an edge image corresponding to the image of the second frame is obtained.

Next, in a potential image generation step of step S23, processing for obtaining a potential image is performed. The potential image is obtained by performing a process of lowering the image energy of the edge portion than the image energy of the other portions based on the edge image obtained in step S22. That is, the potential image is an image in which the potential of the image energy is indicated by the level of the potential.

Next, in the ACM tracking step of step S24, a process of tracking the combined ACM obtained in the process corresponding to the immediately preceding frame to the edge portion of the potential image obtained in step S23. Do.

For the processing, for example, a method in which the ACM has elasticity is used. Ueda et al .: “Contour tracking based on energy minimization”, IEICE (D-2), v
ol. J75, pp. 111-120, (1992-
Use the one shown in 1).

In this case, distance conversion is performed on a potential image showing an edge, and a snake ACM is applied by dynamic programming. The ACM, which is the snake, has elasticity to suppress deformation.

The dynamic programming is, for example, AA AMIN
I Other `` Using Dynamic Programming for Solving Vari
ational Problems in Vision ", IEEE TRANSACTION ON
PATERN ANALYSIS AND MACHINE INTELLIGENCE VOL 12, N
Use the method shown in O. 9, (1990-9).

In such an ACM tracking step, the ACM
Is traced to the edge portion of the potential image, and in an ACM division step of step S25, an ACM satisfying a predetermined division condition is divided.

This ACM division step is performed for the following purpose. That is, when the luminance difference between both sides of the tracked ACM changes abruptly in a certain portion, it is considered that one ACM is applied to two different edges. Therefore, one ACM is split into two ACMs at such a place where the luminance difference changes abruptly.

Next, the processing contents of the ACM division step will be described in detail. FIG. 7 is a schematic diagram illustrating a state of the ACM divided in the ACM division step.

Referring to FIG. 7, A to be subjected to division processing
An arbitrary point D is taken on CM A, and at that point, A
The CM A is bisected into ACM A1 and ACM A2. Then, the luminance difference C on both sides of the ACM A1 is calculated as shown in the following equation (9). At the same time, the luminance difference C2 on both sides of the ACM A2 is calculated as shown by the following equation (10).

[0069]

(Equation 5)

Then, on the ACM A, the luminance difference C1 and
The point at which the difference from the luminance difference C2 is maximized is defined as a split point d. The split point d is represented by the following equation (11).

[0071]

(Equation 6)

Whether or not to divide the thus obtained division point d is determined based on the division condition shown in the following equation (12). In equation (12) below,
ld4 is set to a constant value in advance.

[0073]

(Equation 7)

In this ACM splitting step, AC
When M−A satisfies the above equation (12), the breaking point d
Divides ACM A into two ACMs. This splits ACM A. That is, one ACM
Is split into ACMs that have successfully tracked edges and ACMs that have failed to track, or ACMs that have been applied across two edges.

Next, in the ACM erasure step of step S26, a process for erasing the ACM for which edge tracking has failed is performed. Hereinafter, the contents of the ACM disappearing step will be described in detail.

FIG. 8 is a schematic diagram showing the state of the ACM which is deleted in the ACM deletion step. First, the luminance difference between both sides is calculated over the entire length of ACM A, and the average value C of the luminance difference is calculated. Such a calculation is as follows (1)
This is performed based on equation 3).

[0077]

(Equation 8)

The average value C of the luminance differences thus obtained is
Is an object for determining whether or not ACM A is on an edge on an image. The average value C of the luminance differences thus obtained indicates that ACM A is out of the edge when the value is small. This is because the edge portion has a large difference in luminance between both sides.

Therefore, the average value C of the luminance difference in the ACM for which the tracking has failed becomes equal to or smaller than the fixed value threshold5. For this reason, the following equation (14) is used as an annihilation condition,
ACM A satisfying the condition is extinguished.

[0080]

(Equation 9)

As described above, by the ACM splitting step in step S25 and the ACM extinguishing step in step S26, unnecessary ACMs whose tracking has failed are removed.

Next, in the ACM generation step of step S27, processing for generating a new ACM is performed. The generation of the ACM in this case is based on the edge image obtained in the edge extraction step of step S22, and generates the element of the ACM obtained by the edge. The generation is performed in the same manner as in step S13 described above.

Further, in the ACM generation step, the ACM based on the edge image thus obtained and the ACM remaining after step S26 are coexisted on the image.

Next, in the ACM combining step of step S28, the ACM elements obtained based on the edge image are combined, and the combined ACM is combined with the ACM remaining after the ACM disappearing step. . In such a case, further ACM combining processing is also performed. A new ACM is generated by performing such an ACM combination.

The reason for combining the ACMs in this way is that the ACMs are combined.
Since the ACM that has passed through the erasure step has lost the ACM that failed to be tracked, it is necessary to compensate for the ACM that has been erasured in order to set the ACM used for tracking the edge in the next frame. is there. Therefore, a process of supplementing the lost ACM portion with the ACM obtained based on the edge extraction in the frame is performed.

As described above, the ACM obtained in the ACM combining step in step S28 is the ACM tracking step in the processing corresponding to the next frame (for example, S34).
Used for As described above, a new ACM is also obtained in the processing corresponding to the frame after the first frame. Therefore, even if the ACM fails to track the edge, there is no need for a human to manually set the new ACM.

The processes performed in steps S21 to S28 are similarly performed in subsequent processes (for example, S31 to S38) corresponding to the third and subsequent frames.

By performing the processing as described above, a complicated contour can be tracked in a moving image having a plurality of image frames. It is AC
This is because the setting of M is performed automatically regardless of the person.

Next, the flow of data in the processing from the image input step to the ACM combining step shown in the flowchart of FIG. 2 will be described. That is, the flow of data in the data processing device of FIG. 2 will be described.

FIG. 9 is a block diagram showing the relationship between data processing steps in the data processing device 2 of FIG. 1 and a storage unit for storing data used in each step.

Referring to FIG. 9, the above-described data processing device 2
Includes an input image data section 21, an edge data section 22, a potential image section 23, and an ACM data section 24.

The input image data section 21 is provided with the camera 1 described above.
This is a part for storing data related to the image input from. The edge data section 22 is a section that stores data related to edges of an input image. The potential image unit 23 is a part that stores data related to the potential image. The ACM data section 24 is a section for storing data related to the ACM.

Input image data section 21 to ACM data section 2
Data of each of 4 is appropriately rewritten.

In the input image step of step S 1, data is stored in the input image data section 21. Next, in the edge extraction step of step S2, the input image data unit 21
The edge extraction process is performed based on the data from. Then, the data obtained as a result of the processing is
The data is written to the edge data section 22.

Next, in a potential image generation step of step S3, a potential image generation process is performed based on the data read from the edge data section 22.
Data obtained as a result of the processing is written to the potential image section 23.

Next, in the ACM tracking step of step S4, the potential image section 23 and the ACM data section 24
ACM tracking processing is performed based on the data read from the. Data obtained as a result of the processing is written to the ACM data section 24.

Next, in the ACM division step of step S5, ACM division processing is performed based on the data read from the input image data section 21 and the ACM data section 24, respectively. Data obtained as a result of the processing is written to the ACM data section 24.

Next, in the ACM extinguishing step of step S6, ACM extinguishing processing is performed based on the data read from the input image data unit 21 and the ACM data unit 24, respectively. Data obtained as a result of the processing is written to the ACM data section 24.

Next, in the ACM generation step of step S7, ACM generation processing is performed based on the data read from the edge data section 22 and the data read from the ACM data section 24, respectively. Data obtained as a result of the processing is written to the ACM data section 24.

Next, in the ACM combining step of step S8, ACM combining processing is performed based on the data read from the ACM data section 24. Data obtained as a result of the processing is written to the ACM data section 24. Thus, in the data processing device 2 of FIG. 1, data reading and writing are performed as shown in FIG. 9, so that the process shown in FIG. 2 is executed.

According to this embodiment, the ACM can be automatically set when extracting and tracking the outline in the image. For this reason, the contour of an object that repeats occlusion and appearance, such as a moving image, can be accurately tracked by ACM.

As described above, since it is possible to track a complex contour, in a video communication system or the like, the method for extracting and tracking the contour is effective for recognizing and understanding an object which performs a complicated operation. Can be used.

[0103]

According to the present invention, in the first processing corresponding to the first frame, an edge of an object in an image is extracted, and an active counter model is generated and combined based on the edge information. . Therefore, the initial setting of the active counter model can be automatically performed.

Further, the second frame corresponding to the subsequent frame
Also in the processing of (1), processing for tracking the active counter model and resetting of the active counter model after the tracking are performed. Therefore, the active counter model can be automatically set in the second processing corresponding to the subsequent frame.

As described above, according to the present invention, it is possible to automatically set the active contour model.
Tracking can be performed accurately.

[Brief description of the drawings]

FIG. 1 is a block diagram showing a configuration of an automatic contour extraction / tracking device according to an embodiment of the present invention.

FIG. 2 is a flowchart showing a processing procedure of an automatic contour extraction / tracking method according to an embodiment of the present invention.

FIG. 3 is a schematic diagram showing an edge image obtained in an edge extraction step.

FIG. 4 is a schematic diagram showing a state of a thinned edge image obtained in an ACM generation step.

FIG. 5 is a schematic diagram showing a raw state of an ACM obtained in an ACM generation step.

FIG. 6 is a schematic diagram showing a state of ACMs combined in an ACM combining step.

FIG. 7 is a schematic diagram showing a state of an ACM split in an ACM splitting step.

FIG. 8 is a schematic diagram showing a state of an ACM that is deleted in an ACM deletion step.

9 is a block diagram showing a relationship between data processing steps in the data processing device of FIG. 1 and a data storage unit used in each step.

FIG. 10 is a flowchart showing a processing procedure when a conventional contour tracking method is used for tracking a contour in a moving image.

[Explanation of symbols]

 S12, S22 Edge extraction step S23 Potential image generation step S24 ACM tracking step S25 ACM division step S26 ACM disappearance step S13, S27 ACM generation step S14, S28 ACM connection step

──────────────────────────────────────────────────続 き Continuation of the front page (56) References IEICE Technical Report, September 1995, Vol. 95, No. 279 (PRU95 121-134), p. 25-30 IEICE Technical Report, May 1995, Vol. 95, No. 44 (PRU95 21-36), p. 1-8 (58) Field surveyed (Int.Cl. ⁶ , DB name) G06T 7/20 G06T 9/20 JICST file (JOIS)

Claims (1)

(57) [Claims] 1. A method for extracting and tracking a contour of an image in a plurality of frames obtained by imaging a moving object over time, wherein the contour of the object is extracted and tracked in the image of each frame. A first processing step of performing processing corresponding to a first frame of the plurality of frames; and a plurality of second processing steps of performing processing corresponding to each of a plurality of frames subsequent to the first frame. Processing step, wherein the first processing step comprises: a first edge extraction step of extracting an edge of the object in the images of the plurality of frames; and A first model generating step of generating a plurality of active contour models along the contour; and generating the plurality of active contour models based on a predetermined coupling condition. Combining a first model combining step, wherein each of the plurality of second processing steps includes a second edge extracting step of extracting an edge of the object in an image of a corresponding frame; A potential image generating step of generating a potential image corresponding to the edge based on the information of the above, based on the generated potential image and the active counter model combined in the processing corresponding to the immediately preceding frame, A model tracking step of tracking the active contour model to the contour of the original image; and tracking the contour of the original image in the tracked active contour model. Model splitting step based on A model erasing step of erasing an active contour model of a portion of the original image where the contour tracking has failed, and generating a plurality of active contour models based on edge information extracted in the second edge extracting step. A second model generating step of combining the active contour model successfully tracked with the potential image, and a plurality of active contour models generated in the second model generating step to obtain a contour of the object. A second model combining step of obtaining a plurality of new active contour models along the plurality of active contour models and combining the active contour models based on predetermined combining conditions.