JP4306256B2 - Motion vector detection device, motion vector detection method, program, and recording medium - Google Patents

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to a motion vector detection apparatus, a motion vector detection method, a program for executing a motion vector detection method for detecting a motion vector in a moving image, and a recording medium on which the program is recorded.
[0002]
[Prior art]
Various methods for detecting video motion vectors have been proposed as processing for moving image data, and are used for, for example, subject tracking and camera shake correction in a video camera.
For example, a block matching method, a gradient method, a representative point matching method, and the like are known. The following are mentioned as prior art documents.
[0003]
[Patent Document 1]
JP-A-5-30495
[Patent Document 2]
JP-A-8-237536
[Patent Document 3]
JP-A-6-86149
[0004]
In the block matching method, a representative block is set in the past frame. For example, a representative block is set as an n × n pixel range in the horizontal and vertical directions. Then, in the next frame (current frame), a pixel range corresponding to the representative block is searched, and the direction and distance from the representative block position to the corresponding block in the horizontal / vertical space is used as a motion vector.
In this case, for example, based on the total pixel value of the representative block, an n × n block having substantially the same pixel value total is determined as a corresponding block.
[0005]
The gradient method discriminates corresponding points in the past frame and the current frame using the luminance gradient of the pixel data, and detects a motion vector. For example, Patent Document 1 discloses a motion vector detection device using a gradient method.
[0006]
The representative point matching method is described in, for example, the above Patent Documents 2 and 3.
This representative point matching method will be described with reference to FIG. FIGS. 15A and 15B show a past frame F # 1 and a current frame F # 2 as temporally continuous frames.
As shown in FIGS. 15A and 15B, the attention area ARa is set in the past frame F # 1, and the search area ARs is set in the current frame F # 2. The attention area ARa and the search area ARs have the same size and position.
A pixel at the center of the attention area ARa is set as a representative point r. The relationship between the attention area ARa and the representative point r is enlarged and shown in FIG.
[0007]
After the representative point r is determined in this way, the absolute difference value between the luminance values of all the pixels in the search area ARs and the pixel of the representative point r is calculated in the current frame F # 2. A pixel that gives the smallest difference absolute value among the difference absolute values is set as a candidate point e in the search area ARs.
If the number of candidate points e is one, a vector having the candidate point e as the end point and the representative point r as the start point is set as a motion vector to be obtained. However, in reality, the number of candidate points e is not necessarily one. For example, FIG. 15D shows a case where three candidate points e1, e2, e3 are detected. In such a case, the candidate point closest to the center of the search area ARs (= the position corresponding to the representative point r) is generally used as the end point of the vector. Accordingly, in the case of FIG. 15D, the candidate point e2 is adopted as shown in FIG. 15E, and the motion vector V is obtained with the candidate point e2 as the end point as shown in the figure.
[0008]
[Problems to be solved by the invention]
Each of the above methods has the following problems.
First, in the case of the block matching method, the accuracy of motion vector detection is high, but the corresponding position must be detected in units of blocks, so there is a large amount of calculation processing and the amount of data to be stored. Becomes larger.
Further, it is disadvantageous when there is an enlargement or reduction of the image or a rotational movement. Furthermore, there is a high possibility that a wrong vector is detected when a block extends over a plurality of objects in the image, for example, when a block extends over the background and the object.
[0009]
In the gradient method, the detection error increases where the pixel value changes rapidly. Also, it is easily affected by the value of the luminance gradient due to the noise component.
Furthermore, it is difficult to detect large movements, and hardware attempts to detect large movements. D It is known that a larger scale is required.
[0010]
In the case of the representative point matching method, the hardware is compared with the block matching method. D Although there is an advantage that the scale of the configuration can be reduced, the physical meaning is poor if only the candidate point corresponding to the representative point is the pixel having the smallest pixel difference. In particular, in a portion where the pixel value is flat as the image content, many candidate points are detected, and it is difficult to determine which pixel is the end point of the vector. As described above, for example, a candidate point closest to the representative point is set as a vector end point, but this does not have a clear physical meaning.
In general, the pixel value greatly fluctuates due to a subtle difference in sampling position at a sharp edge of an image. For this reason, matching based on pixel differences tends to cause detection errors.
[0011]
That is, in the conventional motion vector detection method, it is necessary to increase the hardware scale in order to increase the detection accuracy, and conversely, it is necessary to sacrifice the detection accuracy in order to reduce the hardware scale.
[0012]
[Means for Solving the Problems]
Therefore, the present invention provides hardware. D The purpose is to realize accurate motion vector detection without increasing the scale of a.
[0017]
The present invention The motion vector detection device of the present invention compares the representative point setting means for setting a certain pixel in the past frame of the moving image data as a representative point, the pixel data value of the current frame and the pixel data value of the representative point, Comparison determination means for determining pixel data as a candidate point whose correspondence with the representative point is estimated based on the comparison result, and a candidate line segment having the maximum length as a candidate line segment in which the candidate point continues in the scanning direction A line segment determining unit for determining the midpoint of the candidate line segment determined as the maximum line segment by the line segment determining unit, the representative point as a start point, and the midpoint as an end point Vector detecting means for detecting a motion vector.
[0018]
The motion vector detection method of the present invention compares a representative point setting step of setting a certain pixel in a past frame of moving image data as a representative point, a pixel data value of the current frame, and a pixel data value of the representative point. Then, a comparison determination step for determining pixel data as a candidate point whose correspondence with the representative point is estimated based on the comparison result, and the candidate point has a maximum length as a candidate line segment continuous in the scanning direction. A line segment determining step for determining a candidate line segment; a midpoint calculating step for calculating the midpoint of the candidate line segment determined as the maximum line segment in the line segment determining step; and the midpoint as the starting point. And a vector detection step of detecting a motion vector with ending point as an end point.
[0019]
In these apparatuses and methods, the comparison / determination means (comparison / determination step) uses a predetermined pixel range centered on the pixel position corresponding to the representative point in the current frame as a search area, and each pixel in the search area. The data is compared with the pixel data value of the representative point.
The representative point setting means (representative point setting step) sets the pixel of the current frame corresponding to the midpoint calculated by the midpoint calculation means as the representative point for the next frame.
In the motion vector detection device, the comparison determination unit and the line segment determination unit are used to set a plurality of representative points in the past frame of the video image data and detect a motion vector using the representative points. The midpoint calculation means and the vector detection means are provided so as to correspond to each of the plurality of representative points.
[0020]
The program of the present invention is a program for causing the motion vector detection method to be executed by an arithmetic device, and the recording medium of the present invention is a recording medium recording such a program.
[0021]
In the present invention as described above, “center-of-gravity detection” and “line segment detection” are two new algorithms that can perform detection with high accuracy with a small scale of hardware compared to the conventional method as motion vector detection. It is to provide.
“Center-of-gravity detection” detects a pixel close to the pixel value of the representative point as a candidate point with respect to the representative point in the past frame, and detects the center-of-gravity point of the candidate region where the detected candidate points are spatially continuous. . The motion vector is detected with the representative point as the starting point and the center of gravity as the ending point. In this case, the hardware scale is larger than that of “line segment detection”, but highly accurate motion vector detection is possible.
“Line segment detection” detects pixels close to the pixel value of the representative point as a candidate point with respect to the representative point in the past frame, and the candidate point is the longest continuous candidate line segment in the scanning direction (horizontal direction). Detect points. A motion vector is detected with the representative point as the start point and the midpoint as the end point. In this case, the detection accuracy is slightly reduced in “center of gravity detection”, but the hardware scale can be reduced.
[0022]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, as an embodiment of the present invention, motion vector detection as centroid detection and line segment detection will be described. The description will be given in the following order.
1. Motion vector detection by center of gravity detection
1-1 Motion vector detection method by center of gravity detection
1-2 First apparatus configuration example and process for center of gravity detection
1-3 Second Device Configuration Example and Processing Related to Center of Gravity Detection
2. Motion vector detection by line segment detection
2-1 Motion vector detection method by line segment detection
2-2 First apparatus configuration example and processing for line segment detection
2-3 Second apparatus configuration example and processing according to line segment detection
3. Typical point setting method example
4). Use of motion vector detection information
5. Program and recording medium
[0023]
1. Motion vector detection by center of gravity detection
1-1 Motion vector detection method by center of gravity detection
The motion vector detection method based on the center of gravity detection will be described with reference to FIGS.
FIGS. 1A and 1B show a past frame F # 1 and a current frame F # 2 as temporally continuous frames.
As shown in FIGS. 1A and 1B, the attention area ARa is set in the past frame F # 1, and the search area ARs is set in the current frame F # 2. The attention area ARa and the search area ARs have the same size and position.
A pixel at the center of the attention area ARa is set as a representative point r. The relationship between the attention area ARa and the representative point r is enlarged and shown in FIG. ○ indicates a pixel, and among these, the pixel designated as the representative point r is indicated by ●.
Here, as an illustrative example, an area of 7 pixels × 7 pixels in the horizontal / vertical direction is used as the attention area ARa, but in this case, the search area ARs is also an area of 7 pixels × 7 pixels. However, in reality, the attention area ARa and the search area ARs are not necessarily 7 pixels × 7 pixels, and actually, for example, in the horizontal and vertical directions centered on the pixel as the representative point r (± 32). Pixel) × (± 32 pixels).
[0024]
There are various methods for setting the representative point r, which will be described later. Here, the description will be continued assuming that the central pixel of an arbitrary attention area ARa is the first representative point r.
[0025]
After the representative point r is determined in this way, the absolute difference value between the luminance values of all the pixels in the search area ARs and the pixel of the representative point r is calculated in the current frame F # 2. If the obtained difference absolute value is smaller than a certain threshold value, a comparison determination process using the pixel as a candidate point e is performed.
FIG. 1D shows the result of examining the candidate point e by comparing and determining the value of the representative point r for all the pixels in the search area ARs. Here, it is assumed that the candidate point e is detected as shown by the pixel of ●.
[0026]
Here, a region where the candidate points r are spatially (horizontal / vertical) continuous is defined as a candidate region eA. When the candidate point e is detected as shown in FIG. 1D, two candidate areas eA1 and eA2 exist in the search area ARs as shown in FIG.
Thus, when a plurality of candidate areas eA are recognized, the largest candidate area is selected. In the case of FIG. 2A, the candidate area eA1 is selected. If there is one candidate area eA, the candidate area is naturally selected.
Then, the center of gravity point is obtained for the selected candidate area eA1. FIG. 2B shows the center of gravity w.
When the center of gravity point w is obtained, the center of gravity point w is set as the end point of the motion vector. That is, the motion vector to be obtained is a vector having the representative point r as the start point and the center of gravity point w as the end point. As shown in an enlarged view in FIG. 2C, when the pixel position corresponding to the representative point r is ex, the vector of the pixel position ex → centroid point w becomes the motion vector V.
[0027]
The motion vector detection by the center of gravity detection according to the present embodiment is performed by setting the center of gravity in the candidate area eA as the vector end point as described above.
Note that, during a period in which pixel data of a frame F # 3 (not shown) following the frame F # 2 of FIG. 1 is input, the frame F # 2 is a past frame. In that case, the representative point r is set based on the obtained center-of-gravity point w, for example. For example, as the pixel closest to the barycentric point w, the pixel position ey of FIG.
In the frame F # 3, the same position as the attention area ARa is set as the search area ARs, and motion vector detection is performed in the same manner as described above.
[0028]
1-2 First apparatus configuration example and process for center of gravity detection
FIG. 3 shows the configuration of a motion vector detection apparatus that uses the centroid detection method described with reference to FIGS.
As shown in FIG. 3, the motion vector detection device includes a determination unit 1, frame memories 2A and 2B, an address generation unit, and switches SW1 to SW4.
[0029]
The frame memories 2A and 2B are memories that store pixel data Dg forming each frame of the moving image, and are used alternately for each frame. For example, even-numbered frames are stored in the frame memory 2A, and odd-numbered frames are stored in the frame memory 2B.
In the switches SW1 to SW4, for example, the t1 terminal is selected as shown in the figure when the even-numbered frame memory 2A is input as the input pixel data Dg. In the next frame period, switching is performed in units of frame periods such that the t2 terminal is connected.
[0030]
In the frame period in which the switches SW1 to SW4 are connected to the t1 terminal, the input pixel data Dg is supplied to the frame memory 2A via the switch SW1. At this time, the address AD is generated by the address generator 3 based on the clock CK. In other words, the pixel address is a row / column value indicating the pixel position in the horizontal and vertical directions.
This address AD is supplied to the frame memory 2A as an address corresponding to the input pixel data Dg via the switch SW2. The frame memory 2A stores input pixel data Dg based on the address AD.
[0031]
Further, the input pixel data Dg and the address AD are also input to the determination unit 1. The determination unit 1 performs the above-described center-of-gravity detection process on the input pixel data Dg constituting the frame, and detects a motion vector based on the representative point of the previous frame.
[0032]
The determination unit 1 includes a representative point information holding unit 11, a comparison determination unit 12, a threshold holding unit 13, a flag memory 14, a centroid calculation unit 15, a motion vector calculation unit 16, and a motion vector memory 17 as illustrated.
These units can be realized by either software or hardware. For example, when it is configured by hardware, a representative point information holding unit 11, a threshold holding unit 13, a flag memory 14, a register and a memory as a motion vector memory 17 are provided, and a comparison determination unit 12, a centroid calculation unit 15, a motion vector calculation An arithmetic circuit as the unit 16 may be formed.
Further, the determination unit 1 is constituted by an arithmetic processing device such as a DSP or a CPU, and the representative point information holding unit 11, the threshold holding unit 13, the flag memory 14, and the motion vector memory 17 are realized as an internal RAM or a register. The calculation functions as the comparison determination unit 12, the centroid calculation unit 15, and the motion vector calculation unit 16 may be realized by mounting.
[0033]
The representative point information holding unit 11 holds the pixel value and address of the representative point r as information on the representative point r set in the frame before the current input pixel data Dg frame. In the frame period in which the input pixel data Dg is stored in the frame memory 2A as described above, each pixel data of the previous frame is stored in the frame memory 2B. That is, the pixel data as the representative point r is stored in the frame memory 2B. Therefore, the pixel value and address as information of the representative point r are taken from the frame memory 2B via the switch SW3 and held in the representative point information holding unit 11.
The threshold holding unit 13 holds a threshold th as a pixel difference value. The threshold th is preset.
[0034]
The comparison determination unit 12 receives input pixel data Dg and an address AD. The threshold value th is supplied from the threshold value holding unit 13. Further, the pixel value rg of the representative point r and the address rAD of the representative point r are input from the representative point information holding unit 11.
The comparison / determination unit 12 determines whether or not the input pixel data Dg input is within the search area ARs shown in FIG. 1B, and the pixel value of the input pixel data Dg within the search area ARs ( (Luminance level) is compared with the pixel value (luminance level) rg of the representative point r, and determination flag control of the flag memory 14 is performed based on the result.
[0035]
Whether the sequentially supplied input pixel data Dg and its address AD are in the search area ARs is determined by comparing the address AD with the address as the search area ARs.
The search area ARs has the same address range as the attention area ARa centered on the representative point r of the previous frame as described in FIG. For example, FIG. 1 shows an example of 7 × 7 pixels centered on the pixel address of the representative point r for simplification, but actually, for example, in the horizontal and vertical directions centering on the pixel as the representative point r (± 32). Pixel) × (± 32 pixels). In this case, the comparison / determination unit 12 can determine a range of 65 horizontal pixels × 65 vertical pixels centered on the pixel address rAD of the representative point r as the search area ARs.
Thereby, it is possible to determine whether or not the address AD input together with the input pixel data Dg is an address within the range of the search area ARs.
[0036]
When the input pixel data Dg is data in the search area ARs, the input pixel data Dg is compared with the pixel value rg of the representative point r. Specifically, a pixel difference value as a luminance level is calculated. Then, it is determined whether or not the calculated absolute value of the pixel difference value is less than the threshold th.
If it is less than the threshold th, the input pixel data Dg is set as a candidate point e (see FIG. 1D) corresponding to the pixel of the representative point r. In that case, the flag memory 14 is instructed to turn on the determination flag.
[0037]
In the flag memory 14, a flag area corresponding to each pixel for one frame is secured. For example, if the number of horizontal pixels in one frame is Hn, the number of vertical pixels is Vn, and the number of pixels in one frame is Hn × Vn, the flag memory 14 has 1 bit per pixel, A flag area of (Hn × Vn) bits is secured.
When there is a flag ON instruction from the comparison / determination unit 12, a determination flag is set at the bit position corresponding to the address AD supplied at that time. Therefore, a determination flag is set on the flag memory 14 as indicated by the candidate point e in FIG.
It should be noted that for the calculation of the center of gravity described later, the address of the pixel with the flag turned on is required, and an area for storing the address AD corresponding to the flag may be secured, but the flag position itself is the address AD. , It is not necessary to store the address value itself.
[0038]
The center-of-gravity calculation unit 15 refers to the determination flag in the flag memory 14 and selects a candidate area eA (see FIG. 2A) where flag-on addresses (candidate points e) are spatially (horizontal and vertical) continuous. Determine. When there are a plurality of candidate areas eA, the largest candidate area eA is selected.
Then, as described with reference to FIG. 2B, the center of gravity point w is calculated as, for example, a coordinate value for the selected maximum candidate area eA.
[0039]
The motion vector calculation unit 16 calculates a motion vector V based on the centroid point w calculated by the centroid calculation unit 15 and the pixel address rAD of the representative point r. That is, as described with reference to FIG. 2C, the motion vector V is calculated with the coordinate value of the pixel address rAD of the representative point r as the start point and the coordinate value of the barycentric point w as the end point.
The calculated value of the motion vector V, that is, the value indicating the direction and length of the vector is held in the motion vector memory 17 and supplied to a predetermined processing system that performs processing using the motion vector V.
[0040]
For the next frame of the current input pixel data Dg, a representative point r is determined in the current frame. The representative point r is set with reference to the center of gravity point w. For example, the pixel position corresponding to the barycentric point w, for example, the pixel position ey in the example of FIG.
In this way, the centroid calculating unit 15 sets the pixel address rAD ′ as the next representative point r based on the calculated centroid point w, and supplies it to the frame memory 2A via the switch SW4. In the frame memory 2A, information as the representative point r is stored as additional information regarding the pixel as the pixel address rAD ′.
[0041]
Then, when the frame period in which the input pixel data Dg of the next frame is input, the switches SW1 to SW4 are switched to the t2 terminal.
Accordingly, each input pixel data Dg is now stored in the frame memory 2B.
In this frame period, information as the representative point r is read from the frame memory 2A and held in the representative point information holding unit 11.
Similarly to the above, the motion vector V is detected by the center-of-gravity detection method in the processing in the determination unit 1.
Further, when the barycentric point w is calculated in the frame period, the barycentric calculating unit 15 sets the pixel address of the next representative point r based on the calculated barycentric point w. In this case, the pixel address rAD ′ as the next representative point r is supplied to the frame memory 2B via the switch SW4, and information as the representative point r is stored as additional information regarding the pixel as the pixel address rAD ′. The
[0042]
Thereafter, similarly, the input pixel data Dg is stored in the frame memory 2A, and the determination unit 1 uses the information of the representative point r from the frame memory 2B to calculate the motion vector V for the current frame. The process in which the input pixel data Dg is stored in the frame memory 2B and the motion vector V is calculated for the current frame using the information on the representative point r from the frame memory 2A by the determination unit 1 alternately. It will be done.
[0043]
FIG. 4 is a flowchart showing the process of the determination unit 1. This flowchart is shown as processing within one frame period for convenience of explanation. That is, the process of FIG. 4 is performed for each period in which the pixel data Dg in each frame is input.
[0044]
First, representative point information is set as step F101. As described above, the representative point information is the pixel value rg and address rAD as the representative point r set on the previous frame, and is read from the frame memory (2B or 2A) and set in the representative point information holding unit 11. The
Note that the pixel at the representative point r is set based on the barycentric point w in the previous frame, but of course, the motion vector detection is not performed in the very first frame when the motion vector detection is performed. Therefore, the attention area ARa and the representative point r as the center pixel are arbitrarily set on the first frame.
[0045]
Every time the input pixel data Dg is input to the comparison determination unit 12 as Step F102, the processes after Step F103 are performed.
First, in step F103, the comparison determination unit 12 refers to the address AD for the input pixel data Dg, and determines whether the input pixel data Dg is within the search area ARs. If not in the search area ARs, the process returns to step F102 to wait for the next input pixel data Dg.
[0046]
When pixel data Dg within the search area ARs is input, pixel difference calculation is performed in step F104 as processing of the comparison determination unit 12. That is, the absolute difference between the pixel value of the input pixel data Dg and the pixel value of the representative point r is obtained.
In step F105, the pixel difference absolute value is compared with the threshold th. If the pixel difference absolute value is less than the threshold th, the determination flag is turned on for the pixel data Dg on the flag memory 14 in step F106. If the pixel difference absolute value is greater than or equal to the threshold th, the determination flag is not turned on.
[0047]
In step F107, it is determined whether or not the pixel data Dg subjected to the comparison determination is the last pixel in the search area ARs. That is, it is determined whether or not the processing in steps F104, F105, and F106 has been completed for all pixels in the search area ARs. If not completed, the process returns to step F102 to prepare for the next pixel data input. When the pixel data Dg is input, it is similarly determined whether or not it is the search area ARs. If it is within the search area ARs, the processes of steps F104, F105, and F106 are performed.
[0048]
If the comparison determination process is performed for at least all the pixel data Dg in the search area ARs in the current frame, the process proceeds to step F108.
At this time, the information (address) of the candidate point e in the search area ARs is held as a determination flag on the flag memory 14 as shown in FIG. 1D, for example. Therefore, in step F108, the centroid calculation unit 15 refers to the determination flag on the flag memory 14 and performs centroid calculation. That is, the largest candidate area eA is first selected, and the barycentric point w in the candidate area eA is calculated.
[0049]
The barycentric point w is calculated as follows, for example.
FIG. 5 shows the search area ARs where the candidate point e is found as a determination flag, and the x and y coordinate axes thereof are shown.
x coordinate = {Σ (x coordinate of determination flag on pixel)} / (number of flag on pixel)
y coordinate = {Σ (y coordinate of determination flag on pixel)} / (number of flag on pixel)
Calculate as The calculated x-coordinate and y-coordinate are the coordinate values of the barycentric point w.
[0050]
Next, in step F109, based on the calculated center of gravity point w, a representative point r on the current frame for the next frame is set. That is, as described above, the pixel address rAD ′ of the representative point r is set and stored in the frame memory (2A or 2B) storing the pixel data Dg of the current frame.
[0051]
In step F110, the motion vector V is calculated by the motion vector calculator 16 using the coordinate value of the pixel address rAD of the representative point r and the calculated barycentric point w. In step F111, the calculated motion vector V is stored in the motion vector memory 17 and is referred to by a required circuit system.
[0052]
The process of FIG. 4 is executed for each frame period, and the motion vector V is detected for each frame period.
That is, in the motion vector detection by the center-of-gravity detection in this example, pixels close to the representative point r in the past frame are detected as candidate points e, and the detected candidate points e are spatially continuous. The barycentric point w of the candidate area eA to be detected is detected. A motion vector is detected with the representative point r as the starting point and the center of gravity point w as the ending point.
According to this center-of-gravity detection method, it is not a simple point matching between frames, but a vector end point having a physical meaning is detected as the center-of-gravity point of the candidate area eA. Therefore, a highly accurate motion vector can be detected. .
In addition, detecting the barycentric point is less susceptible to the effects of unstable edge portions and noise in the image, and in that sense, accuracy is improved.
Furthermore, the hardware scale may be almost the same as that of the conventional representative point matching method, and the scale can be reduced as compared with the block matching method.
In other words, more accurate motion vector detection is possible with a hardware configuration similar to that of the representative point matching method.
[0053]
In addition, when there are a plurality of candidate areas eA, the centroid calculating unit 15 selects the largest candidate area eA, and calculates the centroid point w for the selected candidate area eA, thereby enabling more accurate vector detection. Become.
Also, by setting a representative point r for the next frame based on the barycentric point w, a physical meaning is given as the representative point r, and the accuracy of motion vector detection can be improved.
[0054]
1-3 Second Device Configuration Example and Processing Related to Center of Gravity Detection
Next, a second apparatus configuration example using the center of gravity detection method will be described.
In the above example, as a basic configuration in the center-of-gravity detection method, one representative point r is set, and pixel comparison determination is performed in the search area ARs corresponding to the attention area ARa including the representative point r.
As an actual application example, not only the case where the number of representative points r is one, but a plurality of representative points r are set in one frame, and each search region corresponds to the attention region ARa including each representative point r. It is also conceivable to perform pixel comparison determination within ARs to detect motion vectors by centroid detection.
[0055]
For example, FIG. 6A shows a state where four representative points r1 to r4 (attention areas ARa1 to ARa4) are set in the frame F # 1, and in this case, the search area ARs is shown in FIG. Thus, in the frame F # 2, four search areas ARs1 to ARs4 are set.
For example, FIG. 7 shows a configuration example when the four representative points r1 to r4 are set in this way. In addition, the same code | symbol is attached | subjected to FIG. 3 and an identical part, and description is abbreviate | omitted.
In FIG. 7, the difference from FIG. 3 is that the determination units 1 </ b> A to 1 </ b> D are provided with four determination units 1 (the same number as the number of representative points) and the determination unit selection unit 4. That is.
[0056]
The determination units 1A to 1D have the same configuration as the determination unit 1 in FIG. That is, each includes a representative point information holding unit 11, a comparison determination unit 12, a threshold holding unit 13, a flag memory 14, a centroid calculation unit 15, a motion vector calculation unit 16, and a motion vector memory 17.
For example, the determination unit 1A is in charge of the search region ARs1, the determination unit 1B is in charge of the search region ARs2, the determination unit 1C is in charge of the search region ARs3, and the determination unit 1D is in charge of the search region ARs4. Therefore, the motion vector V is detected for each search area ARs1 to ARs4.
[0057]
It is the same that each determination unit 1A to 1D can be realized by either software or hardware.
Each of the determination units 1A to 1D may be provided as an independent circuit system (arithmetic processing system). However, for example, a plurality of determination units 1A to 1D referred to here may be provided by one arithmetic processing device such as a DSP or a CPU. Can also be realized.
[0058]
In the case of such a configuration, information (address range) of search areas ARs1 to ARs4 set by the determination units 1A to 1D, respectively, is set in the determination unit selection unit 4. Then, the determination unit selection unit 4 determines whether the address AD generated in synchronization with the input pixel data Dg is an address included in any of the search areas ARs1 to ARs4. When it is determined that the address is in a certain search area ARs, control is performed to process the currently input pixel data Dg in a determination unit (one of 1A to 1D) in charge of the search area ARs. Do.
[0059]
FIG. 8 shows processing for each frame period in the case of such a configuration example. Note that the same processing steps as those in FIG. 4 are denoted by the same step numbers.
The processing of FIG. 8 is executed by each determination unit 1A to 1D.
[0060]
Each determination unit 1A to 1D sets the representative point information for the representative point r in charge on the previous frame in step F101A. That is, each of the determination units 1A to 1D reads the pixel value rg and the address rAD as the representative point r from the frame memory (2B or 2A) and sets them in the representative point information holding unit 11.
Therefore, for example, the representative point information holding unit 11 of the determination unit 1A determines the information of the representative point r1 in FIG. 6A, and the representative point information holding unit 11 of the determination unit 1B determines the information of the representative point r2. Information on the representative point r3 is set in the representative point information holding unit 11 of the unit 1C, and information on the representative point r4 is set in the representative point information holding unit 11 of the determination unit 1D.
At this time, each of the determination units 1A to 1D notifies the determination unit selection unit 4 of information on the search areas ARs1 to ARs4 set according to the representative points r1 to r4.
[0061]
When the input pixel data Dg is input to the comparison / determination unit 12 as step F102, each of the determination units 1A to 1D confirms whether or not the determination unit selection unit 4 has selected at that time in step F103A.
[0062]
The determination unit selection unit 4 selects (process execution instruction) one of the determination units 1A to 1D based on the address AD of the input pixel data Dg (or does not select any of them).
That is, for example, when the address AD of the input pixel data Dg is within the search area ARs1, the determination unit 1A is selected corresponding to the input pixel data Dg.
Similarly, when the address AD of the input pixel data Dg is within the search area ARs2, the determination unit 1B is selected corresponding to the input pixel data Dg.
When the address AD of the input pixel data Dg is within the search area ARs3, the determination unit 1C is selected corresponding to the input pixel data Dg.
When the address AD of the input pixel data Dg is within the search area ARs4, the determination unit 1D is selected corresponding to the input pixel data Dg.
When the address AD of the input pixel data Dg does not correspond to any of the search areas ARs1 to ARs4, none of the determination units 1A to 1D is selected corresponding to the input pixel data Dg.
[0063]
That is, in each of the determination units 1A to 1D, whether or not to process the pixel data Dg input at that time is determined depending on whether or not the determination unit selection unit 4 has selected in step F103A.
If not selected, the process returns to step F102 to wait for the next input pixel data Dg.
If it is selected by the determination unit selection unit 4, the process of steps F104 to F107 is performed on the input pixel data Dg, that is, determination flag control is performed by calculating the pixel difference and comparing the threshold value tH.
Since the processing after Step F104 is the same as that in FIG. 3, the redundant description is avoided. As a result, each of the determination units 1A to 1D obtains the candidate point e in the search areas ARs1 to ARs4 that each handles, The centroid point w is calculated from the maximum candidate area eA, and the motion vector V as the coordinate of the centroid point w is detected from the coordinates of the representative point r.
[0064]
By doing in this way, the motion vector V of each part of the image content can be detected on one screen, and this makes it possible to detect motion vectors with a wide range of usage.
In FIG. 7, the determination unit selection unit 4 is provided. However, the determination units 1 </ b> A to 1 </ b> D are each provided with a function of determining whether the input pixel data Dg is in the search area ARs. It may be.
In this example, the number of representative points r and the search areas ARs is four. However, examples of two, three, or five or more are also conceivable, and for this purpose, the determination unit 1 responds to the number of representative points. What is necessary is just to be comprised.
[0065]
2. Motion vector detection by line segment detection
2-1 Motion vector detection method by line segment detection
Next, a motion vector detection method based on line segment detection will be described with reference to FIG.
FIGS. 9A and 9B show a past frame F # 1 and a current frame F # 2 as temporally continuous frames. As in the case of the center of gravity detection, the attention area ARa is set in the past frame F # 1, and the search area ARs is set in the current frame F # 2. The attention area ARa and the search area ARs have the same size and position. A pixel at the center of the attention area ARa is set as a representative point r. The relationship between the attention area ARa and the representative point r is enlarged and shown in FIG.
[0066]
After the representative point r is determined in this way, the absolute difference value between the luminance values of all the pixels in the search area ARs and the pixel of the representative point r is calculated in the current frame F # 2. If the obtained difference absolute value is smaller than a certain threshold value, a comparison determination process using the pixel as a candidate point e is performed.
FIG. 9D shows the result of examining the candidate point e by comparing and determining the value of the representative point r for all the pixels in the search area ARs. Here, it is assumed that the candidate point e is detected as shown by the pixel of ●.
K, k + 1,... Indicate the scanning direction (horizontal direction) of pixels on the frame. Since the pixel data constituting the frame is input as stream data in the scanning direction, for example, during a certain period in the k-th scanning line period in one frame, the first pixel in the search area ARs shown in the figure. A row of pixels is input.
[0067]
The candidate point e is discriminated as shown in FIG. 9D. In this result, an area where the candidate point r is continuous in the scanning direction (horizontal direction) is set as a candidate line segment eL. When the candidate point e is detected as shown in FIG. 9D, the candidate line segment is located in the search area ARs in the portion where the candidate point e exists as seen for each scanning line as shown in FIG. 9C. eL1, eL2,... eL7 are detected.
Then, the largest (longest) candidate line segment is selected from such candidate line segments.
For example, in the case of FIG. 9E, the candidate line segment eL 2 Is selected. When there is one candidate line segment eL, the candidate line segment is naturally selected.
And the selected candidate Line segment eL2 As shown in FIG. 9F, a midpoint c is obtained. .
When the midpoint c is obtained, the midpoint c is set as the end point of the motion vector. In other words, the motion vector to be obtained is a vector having the representative point r as the start point and the midpoint c as the end point. As shown in FIG. 9 (f), when the pixel position corresponding to the representative point r is ex, the vector from the pixel position ex → the middle point c becomes the motion vector V.
[0068]
The motion vector detection by line segment detection according to the present embodiment is performed so that the midpoint in the candidate line segment eL is the vector end point as described above.
Note that, during a period in which pixel data of a frame F # 3 (not shown) following the frame F # 2 of FIG. 9 is input, the frame F # 2 is a past frame. In this case, the representative point r is set based on the obtained middle point c, for example. For example, as the pixel closest to the middle point c, one of the pixel positions ey1 and ey2 in FIG. 9F is set as the representative point r, and an n × n area around the representative point r is set as the attention area ARa.
In the frame F # 3, the same position as the attention area ARa is set as the search area ARs, and motion vector detection is performed in the same manner as described above.
[0069]
2-2 First apparatus configuration example and processing for line segment detection
FIG. 10 shows the configuration of a motion vector detection apparatus that uses the line segment detection method described in FIG.
As illustrated, the motion vector detection device includes a determination unit 20, frame memories 2A and 2B, an address generation unit, and switches SW1 to SW4.
[0070]
The frame memories 2A and 2B are memories that store pixel data Dg forming each frame of the moving image, and are used alternately for each frame. For example, even-numbered frames are stored in the frame memory 2A, and odd-numbered frames are stored in the frame memory 2B.
In the switches SW1 to SW4, for example, the t1 terminal is selected as shown in the figure when the even-numbered frame memory 2A is input as the input pixel data Dg. In the next frame period, switching is performed in units of frame periods such that the t2 terminal is connected.
[0071]
In the frame period in which the switches SW1 to SW4 are connected to the t1 terminal, the input pixel data Dg is supplied to the frame memory 2A via the switch SW1. At this time, the address AD is generated by the address generator 3 based on the clock CK. In other words, the pixel address is a row / column value indicating the pixel position in the horizontal and vertical directions.
This address AD is supplied to the frame memory 2A as an address corresponding to the input pixel data Dg via the switch SW2. The frame memory 2A stores input pixel data Dg based on the address AD.
[0072]
Further, the input pixel data Dg and the address AD are also input to the determination unit 20. The determination unit 20 performs the above-described line segment detection process on the input pixel data Dg constituting the frame, and detects a motion vector based on the representative point of the previous frame.
[0073]
The determination unit 20 includes a representative point information holding unit 21, a comparison determination unit 22, a threshold holding unit 23, a line segment determination unit 24, a motion vector calculation unit 25, and a motion vector memory 26 as illustrated.
These units can be realized by either software or hardware. For example, when it is configured by hardware, the representative point information holding unit 21, the threshold holding unit 23, a storage unit in a line segment determination unit 24 described later, and a motion vector memory 26 are provided with necessary registers and memories, and are compared and determined. An arithmetic circuit as the unit 22, the line segment determination unit 24, and the motion vector calculation unit 25 may be formed.
Further, the determination unit 20 includes an arithmetic processing device such as a DSP or a CPU, and as an internal RAM or register, a representative point information holding unit 21, a threshold holding unit 23, a storage unit in a line segment determination unit 24 described later, and a motion vector A storage area may be realized as the memory 26, and a calculation function as the comparison determination unit 22, the line segment determination unit 24, and the motion vector calculation unit 25 may be realized by software implementation.
[0074]
The representative point information holding unit 21 holds the pixel value and address of the representative point r as information of the representative point r set in the frame before the current input pixel data Dg frame. In the frame period in which the input pixel data Dg is stored in the frame memory 2A as described above, each pixel data of the previous frame is stored in the frame memory 2B. That is, the pixel data as the representative point r is stored in the frame memory 2B. Therefore, the pixel value and address as information of the representative point r are taken from the frame memory 2B via the switch SW3 and held in the representative point information holding unit 21.
The threshold holding unit 23 holds a threshold th as a pixel difference value. The threshold th is preset.
[0075]
The comparison determination unit 22 receives input pixel data Dg and an address AD. The threshold value th is supplied from the threshold value holding unit 23. Further, the pixel value rg of the representative point r and the address rAD of the representative point r are input from the representative point information holding unit 21.
The comparison determination unit 22 determines whether or not the input pixel data Dg that is input is within the search area ARs shown in FIG. 9B, and the pixel value of the input pixel data Dg within the search area ARs ( The luminance level is compared with the pixel value (luminance level) rg of the representative point r, and the result is notified to the line segment determination unit 24.
[0076]
Whether the sequentially supplied input pixel data Dg and its address AD are in the search area ARs is determined by comparing the address AD with the address as the search area ARs.
The search area ARs has the same address range as the attention area ARa centered on the representative point r of the previous frame as described with reference to FIG. For example, FIG. 9 shows an example of 7 × 7 pixels centered on the pixel address of the representative point r for simplification, but actually, for example, in the horizontal and vertical directions centering on the pixel as the representative point r (± 32). Pixel) × (± 32 pixels). Therefore, in this case, the comparison / determination unit 22 can determine a range of 65 horizontal pixels × 65 vertical pixels centered on the pixel address rAD of the representative point r as the search area ARs.
Thereby, it is possible to determine whether or not the address AD input together with the input pixel data Dg is an address within the range of the search area ARs.
[0077]
When the input pixel data Dg is data in the search area ARs, the input pixel data Dg is compared with the pixel value rg of the representative point r. Specifically, a pixel difference value as a luminance level is calculated. Then, it is determined whether or not the calculated absolute value of the pixel difference value is less than the threshold th.
If it is less than the threshold th, the input pixel data Dg is set as a candidate point e (see FIG. 9D) corresponding to the pixel of the representative point r. In that case, the line segment determination unit 24 is instructed to turn on the measurement flag.
[0078]
The line segment determination unit 24 includes a storage area as a measurement register, an address register, a maximum line segment register, and a midpoint coordinate register, and has a function of performing predetermined arithmetic processing.
The measurement register only needs to hold 1-bit information as a measurement flag.
The address register stores the start point address of the line segment, and may have any number of bits necessary for storing one address AD.
The maximum line segment register is a register that stores the length of the maximum candidate line segment eL, and may be a storage area of the number of bits necessary for the line segment length. The line segment length may simply be information on the number of pixels or the address length. Note that the start point address and end point address of the largest candidate line segment may be stored in the sense of the line segment length.
The midpoint coordinate register stores the midpoint coordinates of the line segment. A storage area that can store coordinate values may be used.
[0079]
When there is an instruction to turn on the flag from the comparison determination unit 12, the line segment determination unit 24 turns on the measurement flag. At that time, the address AD supplied at that time is stored in the address register as the start address of the line segment.
Also, the size (line segment length) of the line segment set as the candidate line segment eL is calculated, and if the value is maximum, it is stored in the maximum line segment register.
Further, the midpoint coordinates are calculated for the candidate line segment eL stored in the maximum line segment register and stored in the midpoint coordinate register.
[0080]
A detailed description will be given in FIG. 11. From this line segment determination unit 24, the maximum candidate line is obtained when the processing of the comparison determination unit 22 and the line segment determination unit 24 is completed for all pixel data Dg in the search area ARs. The midpoint c (midpoint coordinate) for the minute eL is stored in the midpoint coordinate register.
The motion vector calculation unit 25 calculates a motion vector V based on the midpoint c calculated by the line segment determination unit 24 and the pixel address rAD of the representative point r. That is, as described with reference to FIG. 9F, the motion vector V is calculated with the coordinate value of the pixel address rAD of the representative point r as the start point and the coordinate value of the barycentric point w as the end point.
The calculated value of the motion vector V, that is, the value indicating the direction and length of the vector is held in the motion vector memory 17 and supplied to a predetermined processing system that performs processing using the motion vector V.
[0081]
For the next frame of the current input pixel data Dg, a representative point r is determined in the current frame. The representative point r is set with reference to the middle point c. For example, the pixel position corresponding to the middle point c, for example, the pixel position ey1 or ey2 in the example of FIG. 9F is set as the next representative point r.
In this way, the line segment determination unit 24 sets the pixel address rAD ′ as the next representative point r based on the calculated middle point c, and supplies it to the frame memory 2A via the switch SW4. In the frame memory 2A, information as the representative point r is stored as additional information regarding the pixel as the pixel address rAD ′.
[0082]
Then, when the frame period in which the input pixel data Dg of the next frame is input, the switches SW1 to SW4 are switched to the t2 terminal.
Accordingly, each input pixel data Dg is now stored in the frame memory 2B.
In this frame period, information as the representative point r is read from the frame memory 2A and held in the representative point information holding unit 21.
In the same manner as described above, the motion vector V is detected by the midpoint detection method in the processing in the determination unit 20.
Further, when the midpoint c is calculated in the frame period, the line segment determination unit 24 sets the pixel address of the next representative point r based on it. In this case, the pixel address rAD ′ as the next representative point r is supplied to the frame memory 2B via the switch SW4, and information as the representative point r is stored as additional information regarding the pixel as the pixel address rAD ′. The
[0083]
Thereafter, similarly, the input pixel data Dg is stored in the frame memory 2A, and the motion vector V is calculated for the current frame by the determination unit 20 using information on the representative point r from the frame memory 2B. Processing, and the input pixel data Dg is stored in the frame memory 2B, and the determination unit 20 uses the information of the representative point r from the frame memory 2A to calculate the motion vector V for the current frame. Are done alternately.
[0084]
FIG. 11 is a flowchart showing the process of the determination unit 20. This flowchart is shown as processing within one frame period for convenience of explanation. That is, the process of FIG. 11 is performed for each period in which the pixel data Dg in each frame is input.
[0085]
First, representative point information is set as step F201. As described above, the representative point information is the pixel value rg and the address rAD as the representative point r set on the previous frame, and is read from the frame memory (2B or 2A) and set in the representative point information holding unit 21. The
Note that the pixel at the representative point r is set based on the midpoint c in the previous frame. However, naturally, motion vector detection is not performed in the very first frame when motion vector detection is performed. Therefore, the attention area ARa and the representative point r as the center pixel are arbitrarily set on the first frame.
[0086]
Every time the input pixel data Dg is input to the comparison determination unit 22 as Step F202, the processes after Step F203 are performed.
First, in step F203, the comparison / determination unit 22 refers to the address AD for the input pixel data Dg and determines whether the input pixel data Dg is within the search area ARs. If not in the search area ARs, the process returns to step F202 to wait for the next input pixel data Dg.
[0087]
When pixel data Dg within the search area ARs is input, pixel difference calculation is performed in step F204 as processing of the comparison determination unit 22. That is, the absolute difference between the pixel value of the input pixel data Dg and the pixel value of the representative point r is obtained.
In step F205, the pixel difference absolute value is compared with the threshold th. If the pixel difference absolute value is less than the threshold th, the process proceeds to step F207.
And it is discriminate | determined whether the measurement flag in the line segment determination part 24 is turned ON, and if it is OFF, a measurement flag will be turned ON by step F208.
The line segment determination unit 24 turns on the measurement flag in the measurement flag register and stores the current address AD in the address register as the start point address of the line segment in step F209. Then, the process proceeds to Step F210.
If the measurement flag has already been turned on at the time of step F207, the process proceeds to step F210 without performing the processes of steps F208 and F209.
[0088]
In step F210, the line segment determination unit 24 calculates the line segment length as the candidate line segment eL. That is, the line segment length is obtained by subtracting the line segment start point from the line segment end point with the address AD for the current pixel data Dg as the line segment end point, the address stored in the address register as the line segment start point.
In step F211, it is determined whether or not the calculated line segment length is equal to or greater than the line segment length stored in the maximum line segment register at that time.
[0089]
If the calculated line length is equal to or greater than the line length stored in the maximum line segment register at that time, the process proceeds to step F212, and the value of the maximum line segment register is updated to the value calculated this time.
In step F213, the midpoint coordinates of the line segment stored in the maximum line segment register are calculated from the line segment start point address and line segment end point address. In step F214, the value of the midpoint coordinate register is updated to the calculated value. To do. Then, the process proceeds to Step F215.
[0090]
If the pixel difference absolute value is greater than or equal to the threshold th as a comparison result between the pixel difference absolute value and the threshold th in step F205, that is, if the pixel data Dg input this time does not become the candidate point e, the process proceeds to step F206. After resetting the measurement flag, the process proceeds to step F215.
If the line length calculated in step F211 is shorter than the line length stored in the maximum line segment register, the process proceeds to step F215 without updating the maximum line segment register or the midpoint coordinate register. .
[0091]
In each of these cases, when the process proceeds to Step F215, it is determined whether or not the pixel data Dg that is the current processing target is the last pixel in the search area ARs. That is, it is determined whether or not the processing after Step F204 has been completed for all the pixels in the search area ARs. If not completed, the process returns to step F202 to prepare for the next pixel data input. Then, when the pixel data Dg is input, it is similarly determined whether or not it is the search area ARs, and if it is within the search area ARs, the processing after step F204 is performed.
[0092]
Then, when the processing after step F204 is performed on all the pixel data Dg in at least the search area ARs in the current frame, the processing proceeds from step F215 to F216.
At this point, the midpoint coordinates for the maximum length candidate line segment eL are stored in the midpoint coordinate register of the line segment determination unit 24 by the processing described in steps F204 to F214.
[0093]
FIG. 12 schematically shows the processing in steps F204 to F214.
FIG. 12 shows an enlarged view of the case where the candidate point e is detected as shown in FIG. ○ and ● indicate pixels in the frame (input pixel data Dg), and the area within the broken line is the search area ARs. The pixels g1 to g14 indicated by ● are the pixels indicated as the candidate point e in FIG.
[0094]
For example, when pixel data g1 is input in the horizontal period of scan k, this pixel is detected as the first candidate point e. At this time, the measurement flag is turned on, and the address AD of the pixel data g1 is changed to a line segment. Stored in the address register as the starting point.
At this time, the detected line segment length is “1” in terms of the number of pixels since the line segment start point = the line segment end point. Since the candidate line segment eL has not been detected before that, the line segment length “1” becomes the maximum line segment and is stored in the maximum line segment register. The midpoint coordinates are also stored.
[0095]
Next, when the pixel data g2 to be the candidate point e is input, the measurement flag is already on, so the address register is not updated as the line segment start point. At this time, the detected line segment length is “2” in terms of the number of pixels because the line segment start point is the address of the pixel data g1 and the line segment end point is the address of the pixel data g2. At that time, the value of the maximum line segment register is “1”, and since this value is exceeded this time, the value of the maximum line segment register is updated to the line segment length “2”. Further, the value of the midpoint coordinate register is updated to the midpoint coordinates of the line segment by the pixels g1 and g2.
If the next pixel data of the pixel data g2 is input and it does not become the candidate point e, the measurement flag is reset at that time.
[0096]
Thereafter, the input of the pixel data Dg proceeds. For example, when the pixel data g3 is input in the horizontal period of the scan k + 1, and this pixel is detected as the candidate point e, the measurement flag is turned on at this time. The address of the data g3 is stored in the address register as the line segment start point.
At this time, the detected line segment length is “1” in terms of the number of pixels since the line segment start point = the line segment end point. Since the value of the maximum line segment register at that time is “2”, the maximum line segment register and the midpoint coordinate register are not updated.
[0097]
Next, when the pixel data g4 as the candidate point e is input, the measurement register is already on, so the address register is not updated as the line segment start point. At this time, the detected line segment length is “2” in terms of the number of pixels because the line segment start point is the address of the pixel data g3 and the line segment end point is the address of the pixel data g4. At that time, the value of the maximum line segment register is “2”, and this time is equal to or greater than the value of the maximum line segment register, so the value of the maximum line segment register is the line segment length “L” of the candidate line segment eL by the pixels g3 and g4. 2 ", and the value of the midpoint coordinate register is updated to the midpoint coordinates of the line segment of the pixels g3 and g4.
Similarly, when the next pixel data g5 is input and when the next pixel data g6 is input, the maximum value of the line segment is updated, so the values of the maximum line segment register and the midpoint coordinate register are updated. Will be updated. At the time after the pixel data g6 is input, the value of the maximum line segment register is “4”, and the value of the midpoint coordinate register is the midpoint coordinates of the line segments of the pixels g3, g4, g5, and g6. .
If the next pixel data of the pixel data g6 is input and this does not become the candidate point e, the measurement flag is reset at that time.
[0098]
In the case of the example in FIG. 12, thereafter, the calculated line segment length does not become “4” or more even when the pixel data (g7 to g14) as the candidate point e are input. Therefore, the process does not proceed to steps F212 to F214 in FIG. 11, and when the processes after step F204 are completed for all the pixels in the search area ARs, the value of the maximum line segment register is “4” as a result. Thus, the midpoint coordinate register value holds the midpoint coordinates of the line segments of the pixels g3, g4, g5, and g6.
That is, at the time when the processing is completed for all the pixels in the search area ARs in step F215 in FIG. 11, the coordinates of the midpoint c of the candidate line segment eL2 shown in FIGS. Will be held in.
[0099]
When the process proceeds to step F216 in FIG. 11, the line segment determination unit 24 determines the top point of the current frame for the next frame based on the midpoint c of the calculated maximum candidate line segment eL, that is, the value of the midpoint coordinate register. A representative point r at is set. That is, as described above, the pixel address rAD ′ of the representative point r is set and stored in the frame memory (2A or 2B) storing the pixel data Dg of the current frame.
[0100]
In step F217, the motion vector calculation unit 25 calculates the motion vector V using the coordinate value of the pixel address rAD of the representative point r and the midpoint c of the largest candidate line segment eL, that is, the value of the midpoint coordinate register. The In step F218, the calculated motion vector V is stored in the motion vector memory 17 and is referred to by a required circuit system.
[0101]
The process of FIG. 11 is executed for each frame period, and the motion vector V is detected for each frame period.
That is, in the motion vector detection by the line segment detection in this example, a pixel close to the pixel value of the representative point with respect to the representative point r in the past frame is detected as the candidate point e, and the detected candidate point e is detected in the scanning direction. Among the continuous candidate line segments eL, the middle point c of the maximum length candidate line segment eL is detected. Then, the motion vector is detected with the representative point r as the start point and the midpoint c as the end point.
According to this line segment detection method, since the end point of a physically meaningful vector is detected as the midpoint of the largest candidate line segment eL, not just point matching between frames, a highly accurate motion A vector can be detected.
In addition, detecting the midpoint of the line segment is also less susceptible to the effects of unstable edge portions and noise in the image, and in that sense, accuracy is improved.
Also, by setting a representative point for the next frame based on the midpoint, a physical meaning is given as the representative point, and the accuracy of motion vector detection can be improved.
[0102]
The line segment determination unit 24 determines and stores the candidate line segment having the maximum length as the candidate line segment eL in which the candidate points e are continuous in the scanning direction. It is suitable for downsizing of the wear scale.
In other words, the hardware scale, especially the storage area required for processing, can be made smaller than the conventional representative point matching method, and for that reason, more accurate detection is possible than the representative point matching method, which is a simple point-to-point matching method. Become.
[0103]
Also, the line segment detection method is more hardware than the center of gravity detection method described above. D It can be said that it was intended to reduce a.
In the case of the centroid detection method, the centroid w cannot be determined unless the absolute difference value is calculated for all the pixels in the search area ARs and it is determined whether the difference is larger or smaller than the determined threshold value. That is, it is necessary to hold flag information indicating whether or not it can be a candidate point e or data in the search area ARs until the center of gravity w is determined (flag memory in FIG. 3).
On the other hand, in the line segment detection method, it is not necessary to keep information such as the addresses or candidate points of all the pixel data Dg in the search area ARs until the midpoint of the candidate line segment eL to be obtained is determined. .
That is, as described with reference to FIG. 12 above, by adopting a processing form in which the midpoint coordinates are updated each time the maximum line segment is detected, the candidate point e and the candidate line segment related to the past scan are obtained. The eL information is discarded when a longer candidate line segment is detected. Further, information on the candidate point e and the candidate line segment eL that did not exceed the maximum line segment is not stored. Therefore, very little storage space is required to detect the midpoint coordinates, thereby reducing hardware. D It is possible to reduce a.
[0104]
2-3 Second apparatus configuration example and processing according to line segment detection
Next, a second apparatus configuration example using the line segment detection method will be described.
In the example of FIG. 10, as a basic configuration in the line segment detection method, one representative point r is set, and pixel comparison determination is performed in the search area ARs corresponding to the attention area ARa including the representative point r. It was.
Also in this line segment detection method, as a practical application example, not only the case where there is one representative point r, but a plurality of representative points r are set in one frame, and the attention area ARa including each representative point r is set. Correspondingly, it is also conceivable to perform motion vector detection by performing pixel comparison determination within each search area ARs.
[0105]
That is, in the above-described center-of-gravity detection method, as described with reference to FIGS. 6A and 6B, four representative points r1 to r4 (attention areas ARa1 to ARa4) are set in the frame F # 1, and correspondingly, This is a case where four search areas ARs1 to ARs4 are set. For example, FIG. 13 shows a configuration example in which four representative points r1 to r4 are set in this way. In addition, the same code | symbol is attached | subjected to the same part as FIG. 10, and description is abbreviate | omitted.
In FIG. 13, the difference from FIG. 10 in the configuration is that the determination units 20 </ b> A to 20 </ b> D are provided with four determination units 20 (the same number as the set number of representative points) and the determination unit selection unit 4. That is.
[0106]
The determination units 20A to 20D have the same configuration as the determination unit 20 of FIG. That is, each includes a representative point information holding unit 21, a comparison determination unit 22, a threshold holding unit 23, a line segment determination unit 24, a motion vector calculation unit 25, and a motion vector memory 26.
For example, the determination unit 20A is in charge of the search area ARs1 in FIG. 6B, the determination unit 20B is in charge of the search area ARs2, the determination unit 20C is in charge of the search area ARs3, and the determination unit 20D is in charge of the search area ARs4. To do. Therefore, the motion vector V is detected for each search area ARs1 to ARs4.
[0107]
It is the same that each determination unit 20A to 20D can be realized by either software or hardware.
Each of the determination units 20A to 20D may be provided as an independent circuit system (arithmetic processing system). However, for example, a plurality of determination units 20A to 20D referred to here may be performed by one arithmetic processing device such as a DSP or a CPU. Can also be realized.
[0108]
Information (address range) of search areas ARs1 to ARs4 set by the determination units 20A to 20D, respectively, is set in the determination unit selection unit 4. Then, the determination unit selection unit 4 determines whether the address AD generated in synchronization with the input pixel data Dg is an address included in any of the search areas ARs1 to ARs4, and certain search areas ARs If it is determined that the address is the internal address, control is performed for the currently input pixel data Dg to be processed in the determination unit (one of 20A to 20D) in charge of the search area ARs.
[0109]
FIG. 14 shows processing for each frame period in the case of such a configuration example. Note that the same steps as those in FIG. 11 are denoted by the same step numbers.
The process of FIG. 14 is executed by each determination unit 20A to 20D.
[0110]
Each determination unit 20A to 20D sets representative point information for the representative point r in charge on the previous frame in step F201A. That is, each of the determination units 20A to 20D reads the pixel value rg and the address rAD as the representative point r from the frame memory (2B or 2A) and sets them in the representative point information holding unit 21.
Therefore, for example, the representative point information holding unit 21 of the determination unit 20A has information on the representative point r1 in FIG. 6A, and the representative point information holding unit 21 of the determination unit 20B has information on the representative point r2. Information on the representative point r3 is set in the representative point information holding unit 21 of the unit 20C, and information on the representative point r4 is set in the representative point information holding unit 21 of the determination unit 20D. At this time, each of the determination units 20A to 20D notifies the determination unit selection unit 4 of information on the search areas ARs1 to ARs4 set according to the representative points r1 to r4.
[0111]
When the input pixel data Dg is input to the comparison determination unit 22 as Step F202, each determination unit 20A to 20D confirms whether or not the determination unit selection unit 4 has selected at that time in Step F203A.
[0112]
The determination unit selection unit 4 selects (process execution instruction) one of the determination units 20A to 20D based on the address AD of the input pixel data Dg (or does not select any of them).
That is, for example, when the address AD of the input pixel data Dg is within the search area ARs1, the determination unit 20A is selected corresponding to the input pixel data Dg.
Similarly, when the address AD of the input pixel data Dg is within the search area ARs2, the determination unit 20B is selected corresponding to the input pixel data Dg.
When the address AD of the input pixel data Dg is within the search area ARs3, the determination unit 20C is selected corresponding to the input pixel data Dg.
When the address AD of the input pixel data Dg is within the search area ARs4, the determination unit 20D is selected corresponding to the input pixel data Dg.
When the address AD of the input pixel data Dg does not correspond to any of the search areas ARs1 to ARs4, none of the determination units 20A to 20D is selected corresponding to the input pixel data Dg.
[0113]
That is, in each of the determination units 20A to 20D, whether or not to process the pixel data Dg input at that time is determined depending on whether or not the determination unit selection unit 4 has selected in step F203A.
If not selected, the process returns to step F202 to wait for the next input pixel data Dg.
If it is selected by the determination unit selection unit 4, measurement flag control is performed on the input pixel data Dg by the comparison determination unit 22 and the line segment determination unit 24 after step F 204, that is, calculation of the pixel difference and comparison with the threshold value tH. The maximum line segment determination and midpoint calculation are performed.
Since the processing subsequent to step F204 is the same as that in FIG. 11, redundant description is avoided, but as a result, each of the determination units 20A to 20D includes the largest candidate line segment eL in the search areas ARs1 to ARs4 each responsible. Find the point c. Thereafter, in the processing of steps F216 to F218, the motion vector V as the coordinate of the midpoint c is detected from the coordinate of the representative point r.
[0114]
By doing in this way, the motion vector V of each part of the image content can be detected on one screen, and this makes it possible to detect motion vectors with a wide range of usage.
In addition, although it was set as the structure provided with the determination part selection part 4 in FIG. 13, it is provided with the function to discriminate | determine whether each determination part 20A-20D is in the search area | region ARs in charge about each input pixel data Dg. It may be.
In this example, the representative points r and the search areas ARs are set to four. However, an example in which these are set to two, three, or five or more is also conceivable. For this purpose, the determination unit 20 responds to the number of representative points. What is necessary is just to be comprised.
[0115]
3. Typical point setting method example
In the center-of-gravity detection method and line segment detection method described above, the representative point r is set based on the center of gravity w of the candidate area eA of the immediately preceding frame or the midpoint c of the candidate line segment eL of the immediately preceding frame. did. However, since the immediately preceding frame does not exist in the first frame when the motion vector detection is started, the representative point r is arbitrarily set.
For example, the attention area ARa is set on the frame screen, and the central pixel is set as the representative point r. This is the first representative point setting method for the representative point r.
[0116]
Such a first representative point setting method has an advantage that the setting process is simple, while the representative point r is simply the center of the attention area ARs. The meaning of the motion vector is poor, and there is no fear that the accuracy of the motion vector to be determined will be lowered.
Therefore, it is conceivable to set the first representative point r as in the following second and third setting methods.
[0117]
As the second representative point setting method, for example, the center of gravity of the widest flat portion in the attention area ARa is used as the representative point.
This applies the above-described concept of “centroid detection”, that is, a pixel group having similar pixel values in the attention area ARs is detected, and the center of gravity of the pixel group is used as a representative point. When there are a plurality of pixel groups having similar pixel values, the pixel group having the largest area may be selected and the center of gravity point may be set as the representative point r.
For example, if this is applied to the above-described center-of-gravity detection method, motion vectors from the center of gravity to the center of gravity can be detected from the beginning. Further, assuming that the first representative point is meaningful as the “centroid point”, the position change of the first “centroid point” is also followed from the third frame. From these things, the detection accuracy of motion vector detection can be improved.
[0118]
As a third representative point setting method, the inside of the attention area ARa is cut in the horizontal direction, the longest continuous flat part is detected in each horizontal part, and the middle point of the longest continuous line segment in all the horizontal parts Is the representative point.
This applies the above-mentioned concept of “line segment detection”, that is, an arrangement (line segment) of pixels having a small pixel difference as viewed in the horizontal direction (scan line direction) in the attention area AR is detected. The middle point of the longest line segment is designated as the representative point r.
For example, if this is applied to the above-described line segment detection method, the motion vector from the middle point to the middle point can be detected from the beginning. In addition, assuming that the first representative point is meaningful as the “midpoint of the maximum line segment”, the position change of the first “midpoint of the maximum line segment” will be followed after the third frame. . From these things, the detection accuracy of motion vector detection can be improved.
Inspecting the line length in the scanning line direction is Hardware As mentioned earlier, the burden is small.
[0119]
4). Use of motion vector detection information
The motion vector detection according to the embodiment of the present invention has been described above. Here, examples of applications and applications using the motion vector detection result obtained as described above are shown.
For example, the following uses can be considered. Of course, there are uses other than those illustrated.
[0120]
・ Data compression
Optimal compression is performed with reference to the image signal ahead of the motion vector.
・ Object tracking / tracking
For example, the motion of a subject is detected in a video camera, and automatic tracking processing is performed.
Object tracking
For example, a specific target in image data is tracked for image processing or the like.
·Surveillance camera
Track a moving subject in the surveillance camera.
・ Moving object extraction
Extract images of moving objects for camera photography and image processing.
・ Telop extraction
Extract the telop added on the image.
・ Image stabilization
Image stabilization in video cameras
・ Frame double speed
Double the number of frames
・ Slow motion image generation
Create a new frame between frames
・ I / P (interlace / progressive) conversion
・ Noise reduction
Separate motion and noise components
・ High image quality
・ Motion blur correction
Correcting motion blur of moving objects
・ Pixel shift sampling
New information can be obtained by moving objects
・ Camera movement detection
・ Speed detection for in-vehicle cameras
・ Motion prediction
・ 2-3 pull-down detection
[0121]
5. Program and recording medium
The program of the present invention is a program that realizes the motion vector detection processing by the above-described center-of-gravity detection method or line segment detection method by an arithmetic device such as a CPU or DSP.
That is, in the video camera, the image processing apparatus, the personal computer, etc., the processing described in FIGS. 4, 8, 11, and 14 is executed in order to realize the operation of the determination unit 1 by the CPU or the like. The program is the program of the present invention.
[0122]
Furthermore, according to the program recording medium of the present invention on which such a program of the present invention is recorded, the provision of the program becomes easy, and the present invention can be widely provided. For example, it is suitable for designing a video camera and an image processing apparatus, and the present invention can be implemented by a general-purpose computer apparatus.
[0123]
The program of the present invention can be recorded in advance in an HDD (Hard Disk Drive), a ROM, or the like as a recording medium built in the device.
Alternatively, temporarily or permanently stored in a removable recording medium such as a flexible disk, CD-ROM (Compact Disc Read Only Memory), MO (Magnet optical) disk, DVD (Digital Versatile Disc), magnetic disk, semiconductor memory, etc. (Record). Such a removable recording medium can be provided as so-called package software.
In addition to being installed from a removable recording medium, the program of the present invention can also be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.
[0124]
【The invention's effect】
As can be understood from the above description, the present invention has an effect that the motion vector detection can be performed with high accuracy by relatively small scale hardware as “center of gravity detection” and “line segment detection”.
In the centroid detection, pixels close to the pixel value of the representative point with respect to the representative point in the past frame are detected as candidate points, and the centroid point of a candidate region in which the detected candidate points are spatially continuous is detected. A motion vector is detected with the representative point as the starting point and the center of gravity as the ending point.
In line segment detection, pixels that are close to the pixel value of the representative point with respect to the representative point in the past frame are detected as candidate points, and the midpoint of the candidate line segment that has been longest in the scanning direction (horizontal direction) To detect. A motion vector is detected with the representative point as the start point and the midpoint as the end point.
[0125]
In either case, it is not a simple point matching between frames, but a physical meaningful vector end point is detected as the center of gravity of the candidate area or the midpoint of the candidate line segment, so a highly accurate motion vector Can be detected.
In addition, detecting the center of gravity or the midpoint of the line segment is also less susceptible to the effects of unstable edge portions and noise in the image, and in that sense, accuracy can be improved.
The hardware scale may be almost the same as that of the conventional representative point matching method, and the scale can be reduced as compared with the block matching method. Of course, detection with higher accuracy than the representative point matching method, which is simple point-to-point matching, is possible.
[0126]
In the case of centroid detection, the centroid calculation means (centroid calculation step), when there are a plurality of candidate areas, selects the largest candidate area and calculates the centroid point for the selected candidate area. Vector detection is possible.
In the case of line segment detection, the line segment determination means needs to determine and store the candidate line segment having the maximum length as a candidate line segment in which candidate points are continuous in the scanning direction. It is suitable for downsizing the hardware scale.
Also, by setting the representative point for the next frame based on the center of gravity point in the case of center of gravity detection or the midpoint in the case of line segment detection, physical meaning is given as the representative point, and the accuracy of motion vector detection Can be raised.
[0127]
In addition, in order to set a plurality of representative points in the past frame of the moving image data and to detect a motion vector using each of the representative points, in the case of centroid detection, comparison determination means, storage means, centroid calculation means, In the case of line segment detection, the comparison determination unit, the line segment determination unit, the midpoint calculation unit, and the vector detection unit may be provided so as to correspond to each of the plurality of representative points. It is possible to detect motion vectors with a wide range of usage.
[0128]
According to the program of the present invention, a motion vector detection method and a motion vector detection device that achieve the above effects can be realized. According to the recording medium of the present invention, a motion vector detection device that realizes the above effects and a program Suitable for a wide range of offerings.
[Brief description of the drawings]
FIG. 1 is an explanatory diagram of motion vector detection by gravity center detection according to an embodiment of the present invention.
FIG. 2 is an explanatory diagram of motion vector detection by gravity center detection according to the embodiment.
FIG. 3 is a block diagram of a first apparatus configuration example according to the center of gravity detection of the embodiment.
FIG. 4 is a flowchart of motion vector detection processing based on gravity center detection according to the embodiment.
FIG. 5 is an explanatory diagram of a gravity center detection operation according to the embodiment.
FIG. 6 is an explanatory diagram for setting a plurality of representative points according to the embodiment.
FIG. 7 is a block diagram of a second apparatus configuration example according to the center of gravity detection of the embodiment.
FIG. 8 is a flowchart of motion vector detection processing based on gravity center detection according to the embodiment.
FIG. 9 is an explanatory diagram of motion vector detection by line segment detection according to the embodiment of this invention.
FIG. 10 is a block diagram of a first apparatus configuration example according to line segment detection according to the embodiment;
FIG. 11 is a flowchart of motion vector detection processing by line segment detection according to the embodiment.
FIG. 12 is an explanatory diagram of line segment determination processing according to the embodiment;
FIG. 13 is a block diagram of a second apparatus configuration example according to line segment detection according to the embodiment;
FIG. 14 is a flowchart of motion vector detection processing by line segment detection according to the embodiment.
FIG. 15 is an explanatory diagram of a conventional representative point matching method.
[Explanation of symbols]
1, 1A to 1D, 20, 20A to 20D determination unit, 2A, 2B frame memory, 3 address generation unit, 4 determination unit selection unit, 11, 21 representative point information holding unit, 12, 22 comparison determination unit, 13, 23 Threshold holding unit, 14 flag memory, 15 barycenter calculation unit, 16, 25 motion vector calculation unit, 17, 26 motion vector memory, 24 line segment determination unit

Claims (7)

Representative point setting means for setting a certain pixel in a past frame of moving image data as a representative point;
A comparison determination unit that compares the pixel data value of the current frame with the pixel data value of the representative point, and determines pixel data as a candidate point whose correspondence with the representative point is estimated based on the comparison result;
As a candidate line segment in which the candidate points are continuous in the scanning direction, a line segment determining unit that determines a candidate line segment having the maximum length;
Midpoint calculating means for calculating the midpoint of the candidate line segment determined as the maximum line segment by the line segment determining means;
A vector detection means for detecting a motion vector with the representative point as a start point and the midpoint as an end point;
A motion vector detection device comprising: The comparison / determination means uses a predetermined pixel range centered on the pixel position corresponding to the representative point in the current frame as a search area, and for each pixel data in the search area, the pixel data value of the representative point It intends rows comparison motion vector detection device according to claim 1. The representative point setting means, the pixels of the current frame corresponding to the middle point calculated in the mid-point calculating means, a motion vector detection device according to claim 1 shall be the representative point for the next frame. In order to set a plurality of representative points in the past frame of the video image data and detect a motion vector using each of the representative points,
The comparison decision means, the line segment determining means, the mid-point calculating means, and the vector detecting means, a motion vector detection device according to claim 1 that is provided so as to correspond to each of the plurality of representative points. Representative point setting means for setting a certain pixel in a past frame of moving image data as a representative point, the pixel data value of the current frame and the pixel data value of the representative point are compared, and the representative data is based on the comparison result. Comparison determination means for determining pixel data as candidate points whose correspondence with points is estimated, and line segment determination means for determining a candidate line segment having the maximum length as a candidate line segment in which the candidate points are continuous in the scanning direction And a midpoint calculating means for calculating the midpoint of the candidate line segment determined as the maximum line segment by the line segment determining means, and vector detection for detecting a motion vector with the representative point as the start point and the midpoint as the end point A motion vector detection method in a motion vector detection device comprising:
A representative point setting step of setting a certain pixel in a past frame of the moving image data as a representative point;
A comparison determination step of comparing the pixel data value of the current frame with the pixel data value of the representative point and determining pixel data as a candidate point whose correspondence with the representative point is estimated based on the comparison result;
As a candidate line segment in which the candidate points are continuous in the scanning direction, a line segment determination step for determining a candidate line segment having the maximum length;
A midpoint calculating step of calculating a midpoint of the candidate line segment determined as the maximum line segment in the line segment determining step;
A vector detection step of detecting a motion vector with the representative point as a start point and the midpoint as an end point;
Motion vector detection method for performing The representative point setting means for setting a certain pixel in the past frame of the moving image data as a representative point, the pixel data value of the current frame and the pixel data value of the representative point are compared, and the representative data is based on the comparison result. Comparison determining means for determining pixel data as candidate points whose correspondence with points is estimated, and line segment determining means for determining a candidate line segment having the maximum length as a candidate line segment in which the candidate points are continuous in the scanning direction And a midpoint calculating means for calculating the midpoint of the candidate line segment determined as the maximum line segment by the line segment determining means, and vector detection for detecting a motion vector with the representative point as the start point and the midpoint as the end point In a program executed in a motion vector detection device comprising means,
A representative point setting step of setting a certain pixel in a past frame of the moving image data as a representative point;
A comparison determination step of comparing the pixel data value of the current frame with the pixel data value of the representative point and determining pixel data as a candidate point whose correspondence with the representative point is estimated based on the comparison result;
As a candidate line segment in which the candidate points are continuous in the scanning direction, a line segment determination step for determining a candidate line segment having the maximum length;
A midpoint calculating step of calculating a midpoint of the candidate line segment determined as the maximum line segment in the line segment determining step;
A vector detection step of detecting a motion vector with the representative point as a start point and the midpoint as an end point;
A program that executes The representative point setting means for setting a certain pixel in the past frame of the moving image data as a representative point, the pixel data value of the current frame and the pixel data value of the representative point are compared, and the representative data is based on the comparison result. Comparison determining means for determining pixel data as candidate points whose correspondence with points is estimated, and line segment determining means for determining a candidate line segment having the maximum length as a candidate line segment in which the candidate points are continuous in the scanning direction And a midpoint calculating means for calculating the midpoint of the candidate line segment determined as the maximum line segment by the line segment determining means, and vector detection for detecting a motion vector with the representative point as the start point and the midpoint as the end point In a recording medium recording a program executed in a motion vector detection device comprising means,
A representative point setting step of setting a certain pixel in a past frame of the moving image data as a representative point;
A comparison determination step of comparing the pixel data value of the current frame with the pixel data value of the representative point and determining pixel data as a candidate point whose correspondence with the representative point is estimated based on the comparison result;
As a candidate line segment in which the candidate points are continuous in the scanning direction, a line segment determination step for determining a candidate line segment having the maximum length;
A midpoint calculating step of calculating a midpoint of the candidate line segment determined as the maximum line segment in the line segment determining step;
A vector detection step of detecting a motion vector with the representative point as a start point and the midpoint as an end point;
A recording medium on which a program for executing is recorded.

Images