JP7684142B2 - Calculation device, monitoring system, and parallax calculation method - Google Patents

Description

The present invention relates to a computing device, a monitoring system, and a disparity calculation method.

A method is known in which a stereo camera is used to calculate the distance between the camera and an observed object, and a recognition process for the observed object is performed. Patent Document 1 discloses an information processing device that includes: a similarity data generating unit that uses a reference block for each of a plurality of regions obtained by dividing one of left and right captured images captured by a stereo camera that captures the same space from left and right viewpoints, sets a search range in the other image for each reference block, and generates similarity data calculated by associating the image similarity with the reference block with the position within the search range; a similarity correction unit that smoothes the similarity data in the spatial direction based on similarity data generated for a predetermined number of reference blocks surrounding the corresponding reference block; a result evaluation unit that detects the position at which the similarity is maximum in each smoothed similarity data; a depth image generating unit that uses the detection result by the result evaluation unit to calculate the parallax for each reference block, calculates the position of the subject in the depth direction based on the parallax, and generates a depth image by associating it with the image plane; and an output information generating unit that uses the depth image to perform a predetermined information processing based on the position of the subject in three-dimensional space, and outputs the result.

JP 2016-039618 A

The invention described in Patent Document 1 leaves room for improvement in the calculation of parallax.

A calculation device according to a first aspect of the present invention is a calculation device connected to a first imaging unit and a second imaging unit, and generates disparity information between a first image captured by the first imaging unit and a second image captured by the second imaging unit by searching for disparity information in units of a predetermined matching block, and includes a similarity generation unit that generates a first similarity in units of the matching block based on a difference in luminance values between corresponding pixels in a matching block in the first image and a matching block within a search range of the second image, a similarity correction unit that corrects the first similarity using at least one of first correction information generated based on the first similarity of any adjacent matching block within the search range of the second image and second correction information generated based on the disparity information to generate a second similarity, and a similarity determination unit that generates the disparity information based on the second similarity.
A disparity calculation method according to a second aspect of the present invention is a disparity calculation method executed by a calculation device connected to a first imaging unit and a second imaging unit, and configured to generate disparity information between a first image captured by the first imaging unit and a second image captured by the second imaging unit by searching for a predetermined matching block unit, the method including: generating a first similarity for each matching block based on a difference in luminance values between corresponding pixels in a matching block in the first image and a matching block within a search range of the second image; correcting the first similarity using at least one of first correction information generated based on the first similarity of any adjacent matching block within the search range of the second image and second correction information generated based on the disparity information to generate a second similarity; and generating the disparity information based on the second similarity.

The present invention can reduce false matching and improve the accuracy of disparity calculation.

Configuration diagram of a monitoring system including a computing device Similarity correction unit configuration diagram FIG. 1 is a diagram illustrating first similarity information and first correction information; FIG. 1 is a diagram showing a first example of disparity information used to generate second correction information; FIG. 11 is a diagram showing a second example of disparity information used to generate second correction information; FIG. 1 is a diagram illustrating a correction coefficient. FIG. 1 is a diagram for explaining a specific example of a method for generating a correction coefficient. FIG. 13 is a configuration diagram of a correction execution unit according to the second embodiment. FIG. 13 is a configuration diagram of a similarity correction unit according to the third embodiment;

-First embodiment-
A first embodiment of a monitoring system will be described below with reference to FIGS.

Figure 1 is a configuration diagram of a monitoring system S including a calculation device 1. The vehicle surroundings monitoring system S includes a calculation device 1, a first imaging unit 2, a second imaging unit 3, a recognition processing unit 4, and a vehicle control unit 5. The vehicle surroundings monitoring system S is mounted on a vehicle C. The calculation device 1 is connected to the first imaging unit 2, the second imaging unit 3, and the recognition processing unit 4. The recognition processing unit 4 is connected to the calculation device 1 and the vehicle control unit 5.

The calculation device 1, the recognition processing unit 4, and the vehicle control unit 5 are each an ECU (Electronic Control Unit) mounted on the vehicle C. However, the calculation device 1, the recognition processing unit 4, and the vehicle control unit 5 do not have to be individual ECUs, and two or more of them may be realized by the same ECU.

The first and second imaging units 2 and 3 are both cameras, and output captured images obtained by capturing images to the computation device 1. The captured image output by the first imaging unit 2 is called the first captured image 100, and the captured image output by the second imaging unit 3 is called the second captured image 101. The first and second imaging units 2 and 3 are arranged facing the same direction toward the vehicle C, separated by a predetermined baseline length in the horizontal direction. In the following, the horizontal direction in which the first and second imaging units 2 and 3 are lined up is also called the "baseline direction." The computation device 1 generates parallax information 106 based on the first and second captured images 100 and 101 using a method described below, and outputs the parallax information 106 to the recognition processing unit 4.

The calculation device 1 includes an input unit 10, a similarity generation unit 11, a similarity correction unit 12, and a similarity determination unit 13. The input unit 10 is realized by a hardware interface with the first imaging unit 2 and the second imaging unit 3, and by a calculation processing device. The similarity generation unit 11, the similarity correction unit 12, and the similarity determination unit 13 are realized by a calculation processing device. The hardware interface is, for example, RJ45 or Camera Link.

The arithmetic processing device includes, for example, a CPU, which is a central processing unit, a ROM, which is a read-only memory device, and a RAM, which is a readable and writable memory device, and the CPU expands a program stored in the ROM into the RAM and executes it to perform the calculations described below. Instead of a combination of the CPU, ROM, and RAM, the arithmetic processing device may be realized by a rewritable logic circuit such as an FPGA (Field Programmable Gate Array) or an application specific integrated circuit such as an ASIC (Application Specific Integrated Circuit). Instead of a combination of the CPU, ROM, and RAM, the arithmetic processing device may also be realized by a combination of different configurations, such as a combination of the CPU, ROM, RAM, and FPGA.

The input unit 10 supplies the similarity generation unit 11 with a first image 102 obtained by performing image processing on the first captured image 100, and a second image 103 obtained by performing image processing on the second captured image 101. This image processing is performed, for example, using internal parameters and external parameters of the camera, and eliminates adverse effects on the image processing. Examples of image processing include correction of lens distortion and correction of errors in the mounting position and mounting attitude. However, since the processing in the input unit 10 is not essential in this embodiment, the first captured image 100 and the first image 102 may be considered to be the same, and the second captured image 1021 and the second image 103 may be considered to be the same.

The similarity generation unit 11 receives the first image 102 and the second image 103 supplied from the input unit 10, generates first similarity information 104, and supplies it to the similarity correction unit 12. The similarity correction unit 12 generates information representing the similarity between a matching block in the first image 102 and a matching block in the second image 103. The first similarity information 104 is, for example, the luminance variation amount between each pixel in the matching block of the first image 102 and the corresponding pixel in the matching block of the second image 103. The luminance variation amount can be calculated using, for example, SAD (Sum of Absolute Difference), ZSAD (Zero means Sum of Absolute Difference), SSD (Sum of Squared Difference), NCC (Normalized Cross Correlation), ZNCC (Zero means Normalized Cross Correlation), etc.

The first similarity information 104 may be the amount of luminance variation, the direction of luminance variation, or the luminance variation pattern between each pixel in the matching block of the first image 102 and the second image 103 and its surrounding pixels. One example of a luminance variation pattern is an increase/decrease pattern from the left for three horizontal pixels including the left and right adjacent pixels of the target pixel. For example, if the left adjacent pixel value is 10, the target pixel value is 25, and the right adjacent pixel value is 80, the increase/decrease pattern is "increase → increase."

The similarity correction unit 12 generates second similarity information 105 by correcting the first similarity information 104 based on the first similarity information 104 or the disparity information 106. The similarity determination unit 13 detects the position with the highest similarity within the search range based on the second similarity information 105, and generates disparity information 106 based on this position information. The similarity determination unit 13 outputs the generated disparity information 106 to the recognition processing unit 4 and the similarity correction unit 12.

The recognition processing unit 4 performs various recognition processes using the parallax information 106 as input. One example of the recognition process in the recognition processing unit 4 is detection of a three-dimensional object using the parallax information 106. Examples of objects to be recognized by the recognition processing unit 4 include position information, type information, motion information, and danger information of the subject. One example of position information is the direction and distance from the vehicle. One example of type information is pedestrians, adults, children, elderly people, animals, falling rocks, bicycles, surrounding vehicles, surrounding structures, and curbs. One example of motion information is pedestrian or bicycle swaying, jumping out, crossing, movement direction, movement speed, and movement trajectory. One example of danger information is pedestrians jumping out, falling rocks, and abnormal motion of surrounding vehicles such as sudden stops, sudden deceleration, and sudden steering. The recognition information 107 generated by the recognition processing unit 4 is supplied to the vehicle control unit 5.

The vehicle control unit 5 performs various vehicle controls of the vehicle C based on the recognition information 107. Examples of vehicle controls performed by the vehicle control unit 5 include brake control, steering control, accelerator control, vehicle lamp control, warning sound generation, vehicle camera control, and output of information about observed objects around the imaging device to surrounding vehicles and remote center equipment connected via a network. A specific example is speed and brake control according to parallax information 106 of an obstacle present in front of the vehicle.

Distance information may be generated based on the parallax information and used instead of the parallax information 106. Although not described in this embodiment, the vehicle control unit 5 may perform subject detection processing based on the image processing results using the first image 102 and the second image 103. The vehicle control unit 5 may also display images obtained via the first imaging unit 2 or the second imaging unit 3, or images for the viewer to recognize, on a display device connected to the vehicle control unit 5. Furthermore, the vehicle control unit 5 may supply information on observed objects detected based on the image processing results to an information device that processes traffic information such as map information and congestion information.

Figure 2 is a configuration diagram of the similarity correction unit 12. The similarity correction unit 12 includes a first correction information generation unit 20, a second correction information generation unit 21, and a correction execution unit 22. The first correction information generation unit 20 generates first correction information 200 based on the first similarity information 104. The second correction information generation unit 21 generates second correction information 201 based on the disparity information 106. The correction execution unit 22 corrects the first similarity information 104 based on the first correction information 200 and the second correction information 201, and generates second similarity information 105.

The configuration of the first correction information generation unit 20 will be described. The first correction information generation unit 20 includes a correction determination unit 40 and a first generation unit 41. The correction determination unit 40 generates correction validity information 400 indicating whether or not the first correction information 200 needs to be generated, based on the first similarity information 104. For example, the correction determination unit 40 determines that the generation of the first correction information 200 is necessary when the following first determination condition is satisfied, and determines that the generation of the first correction information 200 is unnecessary when the first determination condition is not satisfied.

(Similarity S(i-1) - Similarity S(i)) > TH0 and (Similarity S(i+1) - Similarity S(i)) > TH1

Note that similarity A(i) is the similarity at search position A(i) to be judged within the search range, similarity A(i-1) is the similarity at search position A(i-1) adjacent to the left of search position A(i), and similarity A(i+1) is the similarity at search position A(i+1) adjacent to the right of search position A(i). TH0 and TH1 are predetermined thresholds. Furthermore, the magnitude judgment in the first judgment condition may be changed so that the condition is met when the sizes are equal.

The reason why the first judgment condition refers to the left and right instead of the top and bottom, i.e., the reason why it scans horizontally, is because the baseline direction is horizontal. Also, the first judgment condition may be judged using not only three points but also five points of similarity as follows. However, here, predetermined thresholds TH2 to TH5 are used.

(Similarity S(i-2) - Similarity S(i-1)) > TH2 and (Similarity S(i-1) - Similarity S(i)) > TH3 and (Similarity S(i+1) - Similarity S(i)) > TH4 and (Similarity S(i+2) - Similarity S(i+1)) > TH5

The first generation unit 41 receives the first similarity information 104 and the correction validity information 400 as input, and generates the first correction information 200 based on the correction validity information 400. The first correction information 200 is, for example, a combination of the search position to be corrected and the similarity information after correction, or a combination of a newly generated search position and similarity information. The correction execution unit 22 corrects the similarity by referring to the first correction information 200. The operation of the first correction information generation unit 20 will be described with reference to FIG. 3.

Figure 3 is a diagram illustrating the first similarity information 104 and the first correction information 200. The upper part of Figure 3 shows the entire similarity curve 1010, and the lower part of Figure 3 shows the details of the position portion of the similarity curve 1010. Specifically, the similarity curve 1012 shown in the lower part of Figure 3 indicates the range of the symbol 1011 in the upper part of Figure 3. In the similarity curves 1010 and 1012, the horizontal axis indicates the horizontal search position within the search range, and the vertical axis indicates the similarity, with the similarity being higher the lower on the vertical axis.

The black circles on the similarity curve 1012 shown at the bottom of Figure 3 indicate the similarity at each search position. The similarity generation unit 11 performs a search in units of a specified pixel, so the similarity on the similarity curve 1012 is shown in terms of the pixel unit interval of the search. The pixel unit of the search can be set arbitrarily, but is at most one pixel. The search positions in the center of the figure are 1003, 1001, 1000, 1002, and 1004 from the left.

The similarity S0 at search position 1000 of the similarity curve 1012 is greater than the similarities at the search positions to the left and right. Therefore, search position 1000 satisfies the first judgment condition described above. One example of a method for correcting the similarity curve 1012 is to replace the similarity S0 at search position 1000 with similarity S1. Another example of a method is to generate a new similarity S1 at a search position in the vicinity of search position 1000, for example, search position 1003.

Search position 1003 having similarity S1 in similarity curve 1012 is an example of first correction information 200. One example of a method for generating similarity S1 is a method of calculating it by polynomial interpolation such as linear approximation or SSD parabolic fitting based on multiple similarities in the vicinity. Three similarities of search positions 1000, 1001, and 1002 are examples of multiple similarities that exist in the vicinity. Search position 1003 is a search position with decimal precision that is located between search position 1000 and search position 1002.

In this way, the first correction information generating unit 20 scans in the baseline direction to search for an inflection point where the difference in similarity switches from positive to negative, calculates a new similarity using the similarity around the inflection point, and calculates the position of the inflection point and the combination of the new similarity as the first correction information 200.

When the first correction information 200 includes the search position 1003 and the similarity S1, there are at least three methods for the correction execution unit 22 to correct the similarity using the first correction information 200. The first method is to rewrite the information of the search position 1000 and the similarity S0 with the information of the search position 1003 and the similarity S1. The second method is to leave the information of the search position 1000 and the similarity S0 as is, and to add the information of the search position 1003 and the similarity S1. The third method is to not change the position of the search position 1000, and to rewrite the value of the similarity S0 with the value of S1. When the third method is adopted, the information of the search position 1003 may be calculated by three-point approximation or the like.

Returning to FIG. 2, the configuration of the second correction information generation unit 21 will be described. The second correction information generation unit 21 includes a peripheral parallax continuity determination unit 50 and a second generation unit 51, and generates second correction information 301 based on the parallax information 106 or the parallax boundary information 300.

Each of FIG. 4 and FIG. 5 is a diagram showing an example of disparity information 106 used by the second correction information generating unit 21 to generate the second correction information 201. In the following, since the disparity information 106 is set corresponding to a position, for example, for each block of pixels, the area corresponding to the disparity information 106 used by the second correction information generating unit 21 to generate the second correction information 201 will be referred to as the "reference peripheral disparity area."

Figure 4 is a diagram showing a first example of the disparity information 106 used to generate the second correction information 201. In other words, Figure 4 is a diagram showing a first example of determining a reference peripheral disparity region. Figure 4 shows a process of calculating the disparity of a disparity generation target block 801 in a current frame 800, which is the latest captured image. The grid in Figure 4 is a block that is a unit for calculating the disparity. The size of the block may be one pixel, four pixels, 2 pixels horizontally and 2 pixels vertically, 16 pixels, 4 pixels horizontally and 4 pixels vertically, a square with one side of five or more pixels, or a rectangle composed of any number of pixels.

In the example shown in FIG. 4, the parallax of the upper left block is calculated first, and the processing target is changed in order to the right as indicated by the arrow. After all the parallax in the top row is calculated, the processing target is changed to the second row, and similarly, the processing target is changed in order from left to right. Therefore, when calculating the parallax of the block indicated by the reference symbol 801, all the parallax calculations in the first to third rows are completed, and the calculation of the parallax to the left of the parallax generation target block 801 in the fourth row is also completed. The area indicated by diagonal lines in FIG. 4 is an area of 5 pixels on one side centered on the parallax generation target block 801, in which the parallax has already been calculated at the time of calculating the parallax of the parallax generation target block 801. The parallax information 106 in this area is called the "reference peripheral parallax 802". In the example of FIG. 4, this reference peripheral parallax 802 is used to generate the second correction information 201. That is, in the first example shown in FIG. 4, the reference peripheral parallax area is the reference peripheral parallax 802.

Figure 5 is a diagram showing a second example of the disparity information 106 used to generate the second correction information 201. In other words, Figure 5 is a diagram showing a second example of determining the reference peripheral disparity region. Figure 5 shows three images captured at different times, with the current frame 800 being the latest frame, the first past frame 901 being a frame acquired in the immediately preceding processing cycle, and the second past frame 902 being a frame acquired in the processing cycle two cycles prior. Block 801 of the current frame 800 is the target block for disparity generation, and the corresponding blocks in the past frames are block 903 and block 904.

The disparity of the shaded area in the first past frame 901 is called the "first past reference peripheral disparity 905", and the disparity of the shaded area in the second past frame 902 is called the "second past reference peripheral disparity 906". The first past reference peripheral disparity 905 and the second past reference peripheral disparity 906 are a second example of the disparity information 106 used to generate the second correction information 201. Since these pieces of disparity information are included in past frames, they have already been calculated at the time of generating the disparity generation target block 801, and are disparity information around the blocks 903 and 904 of the past frames corresponding to the disparity generation target block 801. That is, in the second example shown in FIG. 5, the reference peripheral disparity area is only the first past reference peripheral disparity 905, or both the first past reference peripheral disparity 905 and the second past reference peripheral disparity 906. Note that in the example of FIG. 5, two past frames are referenced, but only one past frame or three or more past frames may be referenced.

The peripheral disparity continuity determination unit 50 generates second correction necessity information 500 indicating whether or not it is necessary to generate the second correction information 301 based on the disparity information 106. Examples of the second correction necessity information 500 include peripheral disparity information and correction necessity information.

An example of peripheral disparity information is the average value of disparity values in the reference peripheral disparity region. Examples of methods for calculating this disparity value include the average disparity value of the reference peripheral disparity 802 shown in FIG. 4, the average disparity value of the first past reference peripheral disparity 905 shown in FIG. 5, and the average disparity value of the first past reference peripheral disparity 905 and the second past reference peripheral disparity 906 shown in FIG. 5. Also, disparity values in the horizontal direction only or in the vertical direction only may be referenced. Also, disparities that are isolated within the reference peripheral disparity region may be excluded from the reference target disparity. An example of a method for detecting an isolated disparity is a method for determining that a disparity is isolated when the difference between the value of the detection target disparity and the disparity values surrounding the detection target disparity is greater than a predetermined value.

The second correction necessity information 500 is information indicating whether or not the second correction information 301 is generated in the second generation unit 51. If the second correction necessity information 500 is "Generation: Required", the correction execution unit 22 corrects the similarity using the second correction information 301. If the second correction information 301 is "Generation: Not Required", the correction execution unit 22 does not correct the similarity using the second correction information 301.

For example, the peripheral parallax continuity determination unit 50 calculates the degree of variation of the parallax in the reference peripheral parallax region, and when the degree of variation is large, the second correction necessity information 500 is set to "generation: not required". Examples of the degree of variation include the sum of the differences between the average of the parallax values in the reference peripheral parallax region and each parallax value, and the variance value in the reference peripheral parallax region. The degree of variation may be generated by referring to the parallax values in the horizontal direction only, or the parallax values in the vertical direction only. As another example of a method for generating the correction necessity information, the degree of continuity of the parallax in the reference peripheral parallax region is calculated, and when the degree of continuity is small, the correction necessity information may be set to no correction. Examples of the degree of continuity include the average or variance of the difference values between the values of the parallaxes in the reference peripheral parallax region. The degree of continuity may be generated by referring to the parallax values in the horizontal direction only, or the parallax values in the vertical direction only.

The second generation unit 51 generates the second correction information 301 based on the second correction necessity information 500. An example of the second correction information 301 is a correction coefficient K corresponding to the search position. The role of this correction coefficient K will be explained with reference to FIG. 6. The method of calculating the correction coefficient K will be described later.

Figure 6 is a diagram explaining the correction coefficient K. Graph 1110 shown in the upper part of Figure 6 is an example of the second correction information 301, which is a set of correction coefficients K for each search position. Graph 1111 shown in the lower part of Figure 6 shows the similarity for each search position, and shows the same type of information as Figure 3. However, in Figure 6, the similarity before correction is shown by a solid line, and the correction coefficient after correction is shown by a dashed line. The upper and lower parts of Figure 6 will be explained in order.

In the upper part of FIG. 6, the horizontal axis of graph 1110 indicates the search position within the search range, and the vertical axis indicates the correction coefficient K for the first similarity information 104. Correction coefficient K0 is the value when no correction is made to the first similarity information 104. When the correction coefficient is smaller than correction coefficient K0, the first similarity information 104 is corrected to increase the similarity. When the correction coefficient is larger than correction coefficient K0, the first similarity information 104 is corrected to decrease the similarity.

When the correction coefficient K is used by multiplying it with the first similarity information 104 before correction, for example, the correction coefficient K0 is "1", the correction coefficient K1 is a real number smaller than "1" and larger than "0", and the correction coefficient K2 is a real number larger than "1". When the correction coefficient K is used by multiplying it with the first similarity information 104 before correction, for example, the correction coefficient K0 is "1", the correction coefficient K1 is a real number smaller than "1" and larger than "0", and the correction coefficient K2 is a real number larger than "1". When the correction coefficient K is used by adding it to the first similarity information 104 before correction, for example, the correction coefficient K0 is "0", the correction coefficient K1 is a positive real number, and the correction coefficient K2 is a negative real number.

In the lower part of FIG. 6, the horizontal axis of graph 1111 indicates the search position within the search range, and the vertical axis indicates the similarity. The first similarity curve 1104 shown by a solid line is a similarity curve created using the first similarity information 104 as is. The second similarity curve 1105 shown by a dashed line is a similarity curve created using the second similarity information 105 generated based on the second correction information 1100 and the first similarity information 104.

As shown in graph 1110, the correction coefficient K at search position A1 is "K0" as shown in element 1101 of the second correction information 301. Therefore, as shown in element 1106 on the second similarity curve 1105, the value of the first similarity curve 1104 and the value of the second similarity curve 1105 are the same at search position A1.

The correction coefficient K at search position A2 is "K1", which is a value smaller than K0, as shown in element 1102 of the second correction information 301. Therefore, the second similarity is higher than the first similarity at search position A2, as shown in element 1107 on the second similarity curve 1105. The correction coefficient at search position A3 is "K2", which is a value larger than K0, as shown in element 1103 of the second correction information 301. Therefore, the second similarity is lower than the first similarity at search position A3, as shown in element 1108 on the second similarity curve 1105.
The first similarity information 104 is corrected using the correction coefficient K, for example, as follows.

Sa(Ai) = Sb(Ai)×K(i)

Note that Sb(i) is the similarity at search position (i) on the first similarity curve 1104, Sa(i) is the similarity at search position (i) on the second similarity curve 1105, and K(i) is the correction coefficient K on the second correction information 1100 at search position (i). In this case, the similarity at search position A2 in FIG. 6 has the following relationship.

S1 = S0 x K1

The correction execution unit 22 reads the correction coefficient K for each search position by referring to the second correction information 301, and corrects the first similarity by multiplying it by the similarity corresponding to that position as shown in the above formula. Note that the correction execution unit 22 corrects the similarity using the second correction information 301 after correcting the similarity using the first correction information 200.

A specific example of the generation of the second correction information 301 by the second generation unit 51, specifically, the method of generating the correction coefficient K corresponding to the search position, will be described with reference to FIG. 7. FIG. 7(a) is a diagram showing a first example of calculating the correction coefficient K. The first row shows the disparity position, the second row shows the disparity value, the third row shows the continuity, the fourth row shows the function, and the fifth row shows the correction coefficient K. Here, if the difference with the disparity value adjacent to the left is equal to or less than a predetermined threshold value of "2", it is determined to be continuous and the continuity is increased by "1", and if the difference is equal to or greater than the threshold, the continuity is decreased by "1", with the upper limit of the continuity being set to "3".

Function: "0" means no correction, "1" means weak correction, and "2" means strong correction. When continuity is "0" or "1", the function is set to "0", when continuity is "2", the function is set to "1", and when continuity is "3", the function is set to "2". For convenience of drawing, the correction coefficient K is written as a numerical value different from that in FIG. 6, with "5" corresponding to "K0" in FIG. 6, "3" corresponding to "K1" in FIG. 6, and "4" corresponding to a value between "K1" and "K0" in FIG. 6. The correction coefficient is set according to the value of the function, and when the function is "0", the correction coefficient K is set to "5", when the function is "1", the correction coefficient K is set to "4", and when the function is "2", the correction coefficient K is set to "3".

In the example of FIG. 7(a), the parallax value "8" continues at parallax positions A to G. Therefore, the continuity increases by one from the initial value of "0" to the maximum value of "3". From parallax position H onwards, the parallax value switches to "2", and since the difference from "8" is greater than the threshold value of "2", the continuity decreases by one at parallax positions H, I, and J, and at position J the continuity becomes "0". Since the parallax value "2" continues thereafter, the continuity increases as a continuity of "2" and continues to the maximum value of "3". Since the function is linked to the continuity and the correction coefficient K is linked to the function, the value of the function changes according to the continuity in FIG. 7(a) as well, and the value of the correction coefficient K changes according to the value of the function.

Figure 7 (b) is a diagram showing a second example of calculating the correction coefficient K. The first row shows the disparity position, the second row shows the disparity value, the third row shows continuity A, the fourth row shows continuity B, the fifth row shows the maximum continuity, the sixth row shows the function, and the seventh row shows the correction coefficient K. Continuity A and continuity B count the number of consecutive states of different disparity values. The maximum continuity is the larger value of continuity A and continuity B. In this example, the function is set in conjunction with the maximum continuity.

In the example of FIG. 7(b), disparity positions A through G have a continuous disparity value of "8", disparity positions H through M have a continuous disparity value of "2", and disparity positions N through Q have a continuous disparity value of "5". Therefore, at disparity positions A through G, continuity A increases in the same way as the continuity in FIG. 7(a) and maintains the maximum value of "3". At this time, continuity B has an asterisk indicating no value because there is still only one type of disparity value and it is not used.

When the disparity value switches to "2" at disparity position H, the difference from the previous disparity value "8" is greater than the threshold value "2", so continuity A decreases, becoming the lower limit "0" in this example. Continuity B has an initial value of "0" at disparity position H, and then increases as the disparity value "2" continues, reaching a maximum value of "3" at disparity position K. After that, when the disparity value changes to "5" at disparity position N, continuity B decreases, and continuity A increases in its place. Since the maximum continuity is the maximum value of continuity A and continuity B as described above, the maximum continuity is the same as continuity A up to disparity position G where there is no value of continuity B, and the larger value of continuity B becomes the maximum continuity from disparity position J to N. The value of the function is determined according to the maximum continuity, and the value of the correction coefficient K is determined according to the value of the function.

In the example shown in FIG. 7, the correction coefficient K only takes values corresponding to K0 to K1, but the relationship between the function and the correction coefficient K may be changed as follows so that the correction coefficient K takes values from K0 to K2. For example, when the function is "0", the correction coefficient K is set to "7", i.e., K2; when the function is "1", the correction coefficient K is set to "5", i.e., K0; and when the function is "2", the correction coefficient K is set to "3", i.e., K1.

According to the above-described first embodiment, the following advantageous effects can be obtained.
(1) The arithmetic device 1 is connected to a first imaging unit 2 and a second imaging unit 3, and generates disparity information between a first image 100 captured by the first imaging unit 2 and a second image 101 captured by the second imaging unit 3 by searching for a predetermined matching block unit. The arithmetic device 1 includes a similarity generation unit 11 that generates a first similarity between a matching block in the first image 100 and a matching block in a search range of the second image 101 for each matching block, a similarity correction unit 12 that corrects the first similarity information 104 using at least one of first correction information 200 generated based on the first similarity of any adjacent matching block in the search range of the second image 101 and second correction information 301 generated based on the disparity information 106 to generate second similarity information 105, and a similarity determination unit 13 that generates the disparity information 106 based on the second similarity information 105. Therefore, by correcting the similarity using the first correction information 200 or the second correction information 301, it is possible to prevent erroneous matching and improve the accuracy of calculating the parallax.

(2) The first imaging unit 2 and the second imaging unit 3 generate a first image 100 and a second image 101 at each predetermined processing cycle. As shown in FIG. 4 and FIG. 5, the calculation device 1 includes a second correction information generation unit 21 that generates second correction information, which is a correction coefficient K to be multiplied by the similarity based on the parallax information 106 within the peripheral area of the matching block in the latest first image 100 and the latest second image 101, or the first image 100 from a predetermined cycle ago and the second image 101 from a predetermined cycle ago. Therefore, when the latest captured image is used, the correction coefficient K can be calculated based on the parallax information 106 without positional deviation, and when a previously captured image is used, the correction coefficient can be calculated using the parallax information of the entire frame.

(3) The second correction information 301 includes a correction coefficient that corrects the first similarity corresponding to the disparity information in the surrounding area to be larger when the continuity of the disparity information in the surrounding area is equal to or greater than a threshold, such as search position A2 in FIG. 6. Therefore, by increasing the similarity of a matching block that has a similar disparity to the surrounding area and is expected to be similar, it is possible to prevent erroneous matching and improve the accuracy of disparity calculation.

(4) The second correction information 301 includes a correction coefficient that corrects the first similarity corresponding to the disparity information in the peripheral disparity to be smaller when the continuity of the disparity information in the peripheral region is equal to or smaller than a threshold value, such as search position A3 in FIG. 6. Therefore, by reducing the similarity of a matching block whose disparity is not similar to that of the surroundings, matching becomes more difficult, thereby preventing erroneous matching and improving the accuracy of disparity calculation.

(5) The correction execution unit 22 of the similarity correction unit 12 corrects the first similarity information 104 based on the first correction information 200, and then further corrects the first similarity information 104 based on the second correction information 301 to generate a second similarity. Therefore, the calculation device 1 can use the result of the first correction, which corrects the local similarity, for the second correction, which corrects the wide-area similarity.

(6) The first imaging unit 2 and the second imaging unit 3 are aligned in the baseline direction. A first correction information generating unit 20 is provided, which scans in the baseline direction to search for an inflection point where the difference in similarity switches between positive and negative, calculates a new similarity using the similarity around the inflection point, and calculates the position of the inflection point and the combination of the new similarity as first correction information 200. This makes it possible to perform local correction of similarity at a finer scale than the unit of similarity calculation. For example, if the similarity were calculated for each pixel, the search position 1003 would be a decimal coordinate value, and it can be said that the similarity can be calculated with sub-pixel accuracy.

(7) The surveillance system S includes a calculation device 1, a first imaging unit 2, and a second imaging unit 3. Therefore, the surveillance system S has few mismatches and good parallax calculation accuracy.

(Variation 1)
In the first embodiment described above, the similarity correction unit 12 of the calculation device 1 includes the first correction information generation unit 20 and the second correction information generation unit 21. However, the similarity correction unit 12 may include only one of the first correction information generation unit 20 and the second correction information generation unit 21.

(Variation 2)
In the first embodiment described above, the first imaging unit 2 and the second imaging unit 3 are arranged horizontally, so the baseline direction is horizontal, and the first generation unit 41 scans the similarity in the horizontal direction as shown in Fig. 3. However, the first imaging unit 2 and the second imaging unit 3 may be arranged vertically or diagonally. In this case, the first generation unit 41 may scan the similarity in the baseline direction, i.e., along the epipolar line.

--Second embodiment--
A second embodiment of the monitoring system will be described with reference to Fig. 8. In the following description, the same components as those in the first embodiment are denoted by the same reference numerals, and differences will be mainly described. Points that are not particularly described are the same as those in the first embodiment. This embodiment differs from the first embodiment mainly in that it includes a correction information invalidation unit.

Figure 8 is a configuration diagram of the correction execution unit 22A in the second embodiment. The correction execution unit 22A includes a correction information invalidation unit 60 and a correction unit 61. The correction unit 61 includes a first correction unit 70 and a second correction unit 71.

In the first embodiment, the first similarity information 104, the first correction information 200, and the second correction information 301 are input to the correction execution unit 22, but in the present embodiment, in addition to the above three, an invalidation instruction 602 is further input to the correction execution unit 22A. The correction execution unit 22A performs a correction process on the first similarity information 104 based on the invalidation instruction 602, the first correction information 200, and the second correction information 301, and generates the second similarity information 105.

The correction information invalidation unit 60 can invalidate at least one of the first correction information 200 and the second correction information 301 based on the invalidation instruction 602. For example, the correction information invalidation unit 60 may invalidate both the first correction information 200 and the second correction information 301, may invalidate only one of them, or may invalidate neither. An example of the invalidation instruction 602 is information indicating whether the first correction information 200 is valid or invalid, and information indicating whether the second correction information 301 is valid or invalid. In other words, the correction information invalidation unit 60 accepts the first correction information 200 output by the first correction information generation unit 20 and the second correction information 301 output by the second correction information generation unit 31 as also the second valid correction information 601.

When the correction information invalidation unit 60 receives an invalidation instruction 602 indicating that the first correction information 200 is valid and the second correction information 301 is invalid, the correction information invalidation unit 60 performs, for example, the following process. That is, the correction information invalidation unit 60 outputs the first correction information 200 as the first valid correction information 600, and outputs the second valid correction information 601 as information obtained by invalidating the second correction information 301.

The correction unit 61 performs a correction process on the first similarity information 104 based on the first valid correction information 600 and the second valid correction information 601 to generate the second similarity information 105. As an example of the correction process on the first valid correction information 600, there is a method of generating the second similarity information 105 by replacing the similarity at the search position to be corrected or inserting the similarity at a new search position using the first correction information 200. As an example of the correction process on the second valid correction information 601, there is a method of generating the second similarity curve 1105 described above using FIG. 6. In addition, if the first correction information 200 is invalidated in the first valid correction information 600, the correction process using the first correction information 200 is not performed. If the first correction information 200 is invalidated in the second valid correction information 601, the correction process using the first correction information 200 is not performed.

The first correction unit 70 performs correction processing of the first similarity information 104 based on the first valid correction information 600, and generates intermediate similarity information 700. The second correction unit 71 performs correction processing of the intermediate similarity information 700 based on the second valid correction information 601, and generates second similarity information 105. The correction processing in the first correction unit 70 and the correction processing in the second correction unit 71 are as described using Figures 3 and 6, so details will be omitted.

According to the second embodiment described above, appropriate corrections can be used.

--Third embodiment--
A third embodiment of the monitoring system will be described with reference to Fig. 9. In the following description, the same components as those in the first embodiment are denoted by the same reference numerals, and differences will be mainly described. Points that are not particularly described are the same as those in the first embodiment. This embodiment differs from the first embodiment mainly in that the boundary of the parallax is not detected to generate the second correction information.

Figure 9 is a configuration diagram of the similarity correction unit 12B in the third embodiment. The similarity correction unit 12B includes a second correction information generation unit 21B instead of the second correction information generation unit 21, and also includes a new disparity boundary detection unit 30. The disparity boundary detection unit 30 refers to the calculated disparity information 106, and when it detects a disparity boundary in a matching block, outputs disparity boundary information 300 to the second correction information generation unit 21B. The presence or absence of a disparity boundary in a matching block can be determined, for example, by using any of the following three methods.

The first method is to check the disparity of each block in the matching block in the current frame individually. Specifically, the disparity of each block in the matching block is compared with the disparity of other blocks to the left, right, and above and below, and if the difference is equal to or greater than a predetermined threshold, it is determined that there is a disparity boundary. If the difference between the disparity of any block and the disparity of other blocks to the left, right, and above and below is less than a predetermined threshold, it is determined that there is no disparity boundary. This first method uses the current frame, so the latest information is used, but it cannot evaluate differences with blocks for which disparity has not yet been calculated.

The second method is a method of individually checking the disparity of each block in the matching block in the previous frame acquired immediately before. The difference from the first method is the frame that is the target; since the disparity has already been calculated for all blocks in the previous frame, the second method can calculate the disparity difference even for blocks for which the first method cannot calculate the disparity difference. This second method assumes that there is no significant change in the position of the subject over the course of one frame, and embodies the idea that the benefit of being able to calculate the disparity difference for all blocks outweighs the loss due to changes in the subject's position.

The third method is to detect disparity boundaries using hierarchical search. This method may be applied to the current frame as in the first method, or to past frames as in the second method. In hierarchical search, the disparity is first calculated in the first layer for a block that is larger than the matching block, for example, twice the size of the matching block in both length and width. In what follows, this block is called the large size block.

Next, in the second hierarchical layer, the disparity of each block in the matching block is calculated based on the disparity of the large-sized block. Specifically, a total of three disparities are set as disparity candidates: the disparity of the large-sized block that includes the block for which disparity is to be calculated, and the disparity of the large-sized block adjacent to the left and right of that large-sized block, which is within a predetermined range of the disparity candidates, is set as the disparity to be calculated. Using the disparity calculated in this way by hierarchical search, it is determined whether the difference with the disparity of other blocks to the left, right, above and below is equal to or exceeds a predetermined threshold value, and the presence or absence of a disparity boundary is determined.

When the peripheral disparity continuity determination unit 50 receives c from the disparity boundary detection unit 30, if a disparity boundary exists within the matching block, the peripheral disparity continuity determination unit 50 creates second correction necessity information 500 indicating that generation of the second correction information 301 is unnecessary, and transmits the second correction necessity information 500 to the second generation unit 51. The second generation unit 51 that receives this second correction necessity information 500 does not generate the second correction information 301, and therefore correction of the first similarity information 104 based on the second correction information 301 by the correction execution unit 22 is not performed.

According to the above-described third embodiment, the following advantageous effects can be obtained.
(8) The arithmetic device 1 includes a parallax boundary detection unit 30 that generates parallax boundary information 300 based on luminance information or parallax information of the first image 100 or the second image 101 in the surrounding area. The second correction information generation unit 21 disables generation of the second correction information 201 based on the parallax boundary information 300. Therefore, when a parallax boundary is present in the surrounding area, the similarity correction using the second correction information 301 is not performed, and adverse effects of the similarity correction using the second correction information 301 can be prevented in advance.

(Modification of the third embodiment)
In the third embodiment described above, the parallax boundary detection unit 30 determines the presence or absence of a parallax boundary by using the parallax information 106. However, the parallax boundary detection unit 30 may simply determine the presence or absence of a parallax boundary by utilizing the presence or absence of a luminance change using the first image 102 or the second image 103. The parallax boundary detection unit 30 compares the luminance of each block in the matching block with the luminance of other blocks to the left, right, above, and below with reference to the first image 102 or the second image 103, and determines that a parallax boundary exists if the difference is equal to or greater than a predetermined threshold. If the difference between the luminance of any block and the luminance of other blocks to the left, right, above, and below is less than a predetermined threshold, it determines that there is no luminance boundary.

In each of the above-described embodiments and variations, the functional block configurations are merely examples. Some functional configurations shown as separate functional blocks may be configured together, or a configuration shown in a single functional block diagram may be divided into two or more functions. In addition, some of the functions of each functional block may be provided by other functional blocks.

In each of the above-mentioned embodiments and variations, the program is stored in a ROM (not shown), but the program may be stored in a non-volatile storage device (not shown), such as a flash memory. The arithmetic device may also have an input/output interface (not shown), and the program may be read from another device when necessary via the input/output interface and a medium available to the arithmetic device. Here, the medium refers to, for example, a storage medium that is detachable from the input/output interface, or a communication medium, i.e., a network such as a wired, wireless, or optical network, or a carrier wave or digital signal that propagates through the network. Some or all of the functions realized by the program may be realized by a hardware circuit or an FPGA.

The above-mentioned embodiments and modifications may be combined with each other. Although various embodiments and modifications have been described above, the present invention is not limited to these. Other aspects that are conceivable within the scope of the technical concept of the present invention are also included within the scope of the present invention.

S... monitoring system 1... computing device 10... input unit 11... similarity generation unit 12, 12B... similarity correction unit 13... similarity determination unit 20... first correction information generation unit 21, 21B... second correction information generation unit 22, 22A... correction execution unit 30... parallax boundary detection unit 31... second correction information generation unit 40... correction determination unit 41... first generation unit 50... peripheral parallax continuity determination unit 51... second generation unit 60... correction information invalidation unit 61... correction unit 70... first correction unit 71... second correction unit

Claims (9)

A computing device connected to a first imaging unit and a second imaging unit, the computing device generating disparity information between a first image captured by the first imaging unit and a second image captured by the second imaging unit by searching for a predetermined matching block unit,
a similarity generating unit configured to generate a first similarity on a matching block basis based on a difference in luminance between corresponding pixels in a matching block in the first image and a matching block within a search range of the second image;
a similarity correction unit that corrects the first similarity using at least one of first correction information that is generated based on the first similarity of any adjacent matching blocks within a search range of the second image and second correction information that is generated based on the disparity information to generate a second similarity;
a similarity determination unit that generates the disparity information based on the second similarity. 2. The computing device according to claim 1,
the first imaging section and the second imaging section generate the first image and the second image for each predetermined processing cycle;
a second correction information generation unit that generates the second correction information, which is a correction coefficient to be multiplied by the first similarity, based on the disparity information within a peripheral area of the matching block in a latest first image and a latest second image, or in the first image a predetermined cycle ago and the second image a predetermined cycle ago. 3. The computing device according to claim 2,
The second correction information includes a correction coefficient that corrects the first similarity corresponding to the disparity information in the surrounding area so as to increase the first similarity when the continuity of the disparity information in the surrounding area is equal to or greater than a threshold. 3. The computing device according to claim 2,
The second correction information includes a correction coefficient configured to correct the first similarity corresponding to the disparity information in the surrounding region so as to be smaller when the continuity of the disparity information in the surrounding region is equal to or smaller than a threshold. 3. The computing device according to claim 2,
a parallax boundary detection unit that generates parallax boundary information based on luminance information or the parallax information of the first image in the peripheral region,
The second correction information generation unit disables generation of the second correction information based on the parallax boundary information. 2. The computing device according to claim 1,
A computing device, wherein the similarity correction unit corrects the first similarity based on first correction information, and then further corrects the first similarity based on the second correction information to generate the second similarity. 2. The computing device according to claim 1,
the first imaging unit and the second imaging unit are aligned in a baseline direction,
the first correction information generation unit that scans in the baseline direction to search for an inflection point where the difference in similarity switches between positive and negative, calculates a new similarity using the similarity in the vicinity of the inflection point, and calculates a combination of the position of the inflection point and the new similarity as first correction information. A computing device according to claim 1 ;
The first imaging unit;
The second imaging unit. A disparity calculation method executed by a computing device connected to a first imaging unit and a second imaging unit, the computing device generating disparity information between a first image captured by the first imaging unit and a second image captured by the second imaging unit by searching for a predetermined matching block unit, the disparity calculation method comprising:
generating a first similarity for each matching block based on a difference in luminance between corresponding pixels in the matching block in the first image and a matching block within a search range of the second image;
correcting the first similarity using at least one of first correction information generated based on the first similarity of any adjacent matching block within a search range of the second image and second correction information generated based on the disparity information to generate a second similarity;
generating the disparity information based on the second similarity.

Images