JP7516204B2 - IMAGE PROCESSING DEVICE, STEREO CAMERA DEVICE, MOBILE BODY, AND IMAGE PROCESSING METHOD - Google Patents

Description

The present invention relates to an image processing device, a stereo camera device, a moving object, and an image processing method.

It is known that the position of an object in real space can be calculated based on the difference in the position (parallax) of the same object in each of the stereo images captured by a stereo camera. To calculate the parallax, a so-called block matching method has been proposed in which a small area of a corresponding comparison image is shifted by one pixel at a time in the horizontal direction relative to a small area of a reference image, and the pixel shift amount at which the city block distance becomes the minimum value is calculated (see Patent Document 1).

Patent Publication 2000-207965

However, the block matching method can sometimes result in reduced disparity accuracy.

Therefore, in consideration of the problems with the conventional technology described above, the objective of the present disclosure is to provide an image processing device, a stereo camera device, a moving object, and an image processing method that improve the accuracy of calculated disparity.

In order to solve the above-mentioned problems, an image processing apparatus according to a first aspect comprises:
An acquisition unit that acquires a standard image and a reference image captured by a stereo camera;
a control unit that creates a first fitting function indicating a relationship between the degree of disparity between the search area and the comparison area with respect to an amount of movement of the comparison area in the baseline length direction, based on a plurality of degrees of disparity between a search area in the standard image and a comparison area that is displaced in the reference image by a pixel unit along a baseline length direction of the stereo camera, and selects, based on at least one of an extreme value of the first fitting function and a slope near the extreme value, one of a disparity calculated based on the first fitting function and a disparity calculated using dynamic programming, as a disparity of a subject image in the search area.

Moreover, a stereo camera device according to a second aspect of the present invention comprises:
A stereo camera that captures a standard image and a reference image;
and an image processing device that creates a first fitting function indicating a relationship between the degree of disparity between the search area and the comparison area with respect to an amount of movement of the comparison area in the baseline length direction, based on a plurality of degrees of disparity between a search area in the standard image and a comparison area that is displaced in the reference image by a pixel unit along a baseline length direction of the stereo camera, and selects, based on at least one of an extreme value of the first fitting function and a slope near the extreme value, one of a disparity calculated based on the first fitting function and a disparity calculated using dynamic programming, as a disparity of a subject image in the search area.

A moving body according to a third aspect of the present invention comprises:
The stereo camera device includes: a stereo camera that captures a standard image and a reference image; and an image processing device that creates a first fitting function that indicates a relationship between the degree of dissimilarity of the search area and the comparison area relative to an amount of movement of the comparison area in the baseline length direction based on a plurality of degrees of dissimilarity between a search area in the standard image and a comparison area that is displaced in the reference image on a pixel-by-pixel basis along a baseline length direction of the stereo camera, and selects, as a dissimilarity of a subject image in the search area, one of a dissimilarity calculated based on the first fitting function and a dissimilarity calculated using dynamic programming based on at least one of an extreme value of the first fitting function and a slope near the extreme value.

Further, an image processing method according to a fourth aspect comprises:
A step of capturing a standard image and a reference image by a stereo camera;
creating a first fitting function that indicates a relationship between the degree of difference between the search area and the comparison area relative to an amount of movement of the comparison area in the baseline length direction, based on a plurality of degrees of difference between the search area in the standard image and a comparison area that is displaced in the reference image in units of pixels along a baseline length direction of the stereo camera;
and selecting, based on at least one of an extreme value of the first fitting function and a slope near the extreme value, either the disparity calculated based on the first fitting function or the disparity calculated using dynamic programming as the disparity of the subject image within the search area.

The image processing device, stereo camera device, moving body, and image processing method according to the present disclosure configured as described above improve the accuracy of parallax.

1 is a block diagram showing a schematic configuration of a stereo camera device including an image processing device according to an embodiment of the present invention. 2 is a side view conceptually illustrating the position of the stereo camera of FIG. 1 on the moving body. 2 is a front view conceptually illustrating the position of the stereo camera of FIG. 1 on the moving body. 4 is a flowchart for explaining a position calculation process executed by a control unit in FIG. 1 . 13 is a graph showing a case in which there is a relatively large deviation between the amount of movement of the extreme value of the first fitting function and the amount of movement of the extreme value of a fitting function in which the amount of movement is made infinitely small. 13 is a graph showing a case in which the deviation between the amount of movement of the extreme value of the first fitting function and the amount of movement of the extreme value of a fitting function in which the amount of movement is infinitely small is relatively small. 11 is a graph showing the relationship between the shift amount deviation of each of the first fitting function and the second fitting function and the respective extreme values.

Below, an embodiment of an image processing device to which the present disclosure is applied will be described with reference to the drawings.

As shown in FIG. 1, a stereo camera device 11 including an image processing device 10 of this embodiment includes a stereo camera 12 and an image processing device 10. As shown in FIG. 2, the stereo camera device 11 calculates the position of a subject corresponding to the subject image included in the stereo image in a world coordinate system based on a moving body 13, based on a captured stereo image. As shown in FIG. 1, the stereo camera device 11 notifies an external device 14 of the position of the subject in the world coordinate system. The external device 14 includes a driving support device such as an auto cruise control, a driving support device such as an automatic braking device, a notification device that notifies the distance to the subject, and a warning device that issues a warning.

The moving body 13 may include, for example, vehicles, ships, aircraft, etc. The vehicles may include, for example, automobiles, industrial vehicles, railroad vehicles, residential vehicles, fixed-wing aircraft running on runways, etc. The automobiles may include, for example, passenger cars, trucks, buses, motorcycles, trolley buses, etc. The industrial vehicles may include, for example, industrial vehicles for agriculture and construction. The industrial vehicles may include, for example, forklifts, golf carts, etc. The agricultural industrial vehicles may include, for example, tractors, cultivators, transplanters, binders, combines, lawnmowers, etc. The construction industrial vehicles may include, for example, bulldozers, scrapers, excavators, cranes, dump trucks, road rollers, etc. The vehicles may include those that run by human power. The classification of vehicles is not limited to the above examples. For example, automobiles may include industrial vehicles that can run on roads. The same vehicle may be included in multiple classifications. The ships may include, for example, marine jets, boats, and tankers. Aircraft may include, for example, fixed-wing aircraft, rotorcraft, etc.

The stereo camera 12 and the image processing device 10 can communicate with each other by wired or wireless communication. The stereo camera 12 and the image processing device 10 may communicate with each other via a network. The network may include, for example, a wired or wireless LAN (Local Area Network), CAN (Controller Area Network), etc. The stereo camera 12 and the image processing device 10 may be housed in the same housing and configured as an integrated unit. The stereo camera 12 and the image processing device 10 may be located in a mobile object 13 and configured to be able to communicate with an ECU (Electronic Control Unit) in the mobile object 13.

As shown in FIG. 2, the stereo camera device 11 is mounted on, for example, a moving body 13. The stereo camera 12 is installed, for example, so as to capture an image in front of the moving body 13. The stereo camera 12 is arranged, for example, so that the optical axis OX of at least one of the multiple cameras included in the stereo camera 12, which will be described later, is approximately parallel to the front-rear direction of the moving body 13.

The stereo camera 12 may be mounted in various locations on the mobile body 13. For example, the stereo camera 12 may be mounted inside the mobile body 13 and capture images of the outside of the mobile body 13 through the windshield. The stereo camera 12 may be positioned in front of the rearview mirror or on the dashboard. The stereo camera 12 may be fixed to any of the front bumper, fender grille, side fender, light module, and bonnet of the mobile body 13.

The stereo camera 12 includes multiple cameras having a parallax with respect to each other. As shown in FIG. 1, the stereo camera 12 includes, for example, a first camera 15 and a second camera 16. The stereo camera 12 can capture images of an object from multiple directions by using multiple cameras in cooperation with each other. The stereo camera 12 may include multiple cameras in a single housing. The stereo camera 12 may include two or more cameras that are independent of each other and positioned apart from each other. The stereo camera 12 is not limited to multiple cameras that are independent of each other. In the present disclosure, for example, a camera having an optical mechanism that guides light incident at two separate points to one light receiving element may be adopted as the stereo camera 12.

In the stereo camera 12, the first camera 15 and the second camera 16 are positioned so that their optical axes OX face in directions that allow them to capture the same subject. The optical axes OX and positions of the first camera 15 and the second camera 16 are determined so that the captured images include at least the same subject image. The optical axes OX of the first camera 15 and the second camera 16 are, for example, parallel to each other. The optical axes OX being parallel is not limited to being strictly parallel, and may allow for misalignment in assembly, misalignment in installation, and misalignment over time. The optical axes OX of the first camera 15 and the second camera 16 are not limited to being parallel, and may face in different directions.

The first camera 15 and the second camera 16 are positioned apart in a direction intersecting the optical axis OX. As shown in FIG. 3, the first camera 15 and the second camera 16 are positioned along the left-right direction in a posture determined when they are attached to the moving body 13. Due to differences in the positions of the first camera 15 and the second camera 16, the positions of the corresponding subjects in the two images captured by each camera are different. The base image generated by the first camera 15 and the reference image generated by the second camera 16 are stereo images captured from different viewpoints. The first camera 15 and the second camera 16 capture images of the subject at a predetermined frame rate (e.g., 30 fps).

The first camera 15 and the second camera 16 each include an optical system that defines an optical axis OX and an imaging element. The first camera 15 and the second camera 16 each have a different optical axis OX. The optical system includes a lens or a mirror, and forms an image of a subject on the light receiving surface of the imaging element. The imaging element is, for example, a CCD image sensor (Charge-Coupled Device Image Sensor) or a CMOS image sensor (Complementary MOS Image Sensor). The imaging elements of the first camera 15 and the second camera 16 may exist in the same plane perpendicular to the optical axis OX of each camera. The imaging elements of the first camera 15 and the second camera 16 each have a plurality of pixels arranged in a matrix. In the imaging elements of the first camera 15 and the second camera 16, the row direction in which the pixels are arranged may be parallel to the baseline length direction of the stereo camera 12. The baseline length direction is the direction of a straight line passing through the intersection of the optical axis OX of the first camera 15 with the light receiving surface of the imaging element of the second camera 16 and the intersection of the optical axis OX of the second camera 16 with the light receiving surface of the imaging element. The first camera 15 and the second camera 16 generate signals based on the images formed on the imaging elements. The first camera 15 and the second camera 16 may perform any image processing such as distortion correction, brightness adjustment, contrast adjustment, and gamma correction on the captured images.

As shown in FIG. 1, the image processing device 10 includes an acquisition unit 17, an output unit 18, a memory 19, and a control unit 20. The image processing device 10 may be disposed at any position within the mobile object 13. The image processing device 10 is disposed, for example, within the dashboard of the mobile object 13.

The acquisition unit 17 is an input interface of the image processing device 10 that acquires information from the stereo camera 12 and external devices. The acquisition unit 17 employs a physical connector and a wireless communication device. The physical connector includes an electrical connector that supports transmission by electrical signals, an optical connector that supports transmission by optical signals, and an electromagnetic connector that supports transmission by electromagnetic waves. The electrical connector includes a connector that complies with IEC 60603, a connector that complies with the USB standard, a connector that supports an RCA terminal, a connector that supports an S terminal defined in EIAJ CP-1211A, a connector that supports a D terminal defined in EIAJ RC-5237, a connector that complies with the HDMI (registered trademark) standard, and a connector that supports a coaxial cable including BNC. The optical connector includes various connectors that comply with IEC 61754. The wireless communication device includes a wireless communication device that complies with each standard including Bluetooth (registered trademark) and IEEE 802.11. The wireless communication device includes at least one antenna.

The acquisition unit 17 acquires the standard image and the reference image generated by the stereo camera 12 through imaging as signals. The acquisition unit 17 passes the acquired standard image and reference image to the control unit 20. The acquisition unit 17 may be compatible with a transmission method for the imaging signal of the stereo camera 12. The acquisition unit 17 may be connected to the output interface of the stereo camera 12 via a network.

The output unit 18 is an output interface of the image processing device 10 that can output the position of the subject in the world coordinate system calculated by the image processing device 10 to other devices within the mobile body 13 or to external devices 14 outside the mobile body 13, such as other mobile bodies or roadside units. The output unit 18, like the acquisition unit 17, includes various interfaces compatible with wired and wireless communication. The output unit 18 has, for example, a CAN interface and communicates with other devices within the mobile body 13.

The memory 19 includes any storage device, such as a RAM (Random Access Memory) and a ROM (Read Only Memory). The memory 19 stores various programs that cause the control unit 20 to function, and various information used by the control unit 20.

The control unit 20 includes one or more processors and memories. The processor may include a general-purpose processor that loads a specific program to execute a specific function, and a dedicated processor specialized for a specific process. The dedicated processor may include an application specific integrated circuit (ASIC). The processor may include a programmable logic device (PLD). The PLD may include a field-programmable gate array (FPGA). The control unit 20 may be either a system-on-a-chip (SoC) in which one or more processors work together, or a system in a package (SIP).

The control unit 20 creates at least a first fitting function as described below. The control unit 20 designates an arbitrary region in the reference image as a search region. For example, the control unit 20 designates an arbitrary pixel as a pixel of interest and designates an area of a predetermined size including the pixel of interest as the search region. The search region may be, for example, an area of m rows and n columns (m and n are natural numbers) centered on the pixel of interest.

The control unit 20 designates, as a comparison region, a region in the reference image that is displaced in pixel units from the position of the search region along the baseline length direction. The control unit 20 designates multiple comparison regions by displacing one pixel at a time in both directions along the baseline length direction.

The control unit 20 calculates the degree of difference between the partial images included in the search area and the comparison area. The degree of difference is an index that changes according to the difference in pixel values of the pixels that make up the partial image, such as SAD (Sum of Absolute Difference) and SSD (Sum of Squared Difference). The control unit 20 calculates multiple degrees of difference for each displaced position. In this embodiment, the degree of difference increases as the visual difference between the search area and the comparison area increases, such as SAD and SSD.

In general, a comparison area with an extreme dissimilarity is the most similar to the search area among the comparison areas for which the dissimilarity was calculated. In this embodiment, the extreme dissimilarity is a minimum value in a configuration such as SAD and SSD in which the dissimilarity increases as the visual difference between the search area and the comparison area increases.

The control unit 20 creates a first fitting function based on the dissimilarity calculated for each of the multiple comparison regions. The first fitting function is a function that indicates the relationship between the dissimilarity and the amount of movement of the comparison region in the baseline length direction relative to the search region.

The control unit 20 may create at least one of a first coarse function and a first detailed function as the first fitting function. The first coarse function may be a function in which two dissimilarity degrees whose movement amounts are adjacent to each other are connected by a line segment. The first detailed function is created by performing equiangular linear fitting or parabolic fitting so that the function passes through the dissimilarity degree corresponding to the aforementioned movement amount.

The control unit 20 may further create a second fitting function, as described below. The control unit 20 displaces one of the base image and the reference image by 0.5 pixels in the baseline length direction. Displacing by 0.5 pixels means, for example, performing pixel interpolation by averaging the pixel values of pixels that make up the image adjacent to each other along the displacement direction, and calculating the pixel value at the position displaced by 0.5 pixels. The displacement of one of the base image and the reference image may not be the entire image, but may be the periphery of the search area and comparison area where the parallax is calculated. The displacement direction may be a negative direction of -0.5 pixels.

The control unit 20 displaces one of the base image and the reference image by 0.5 pixels in the baseline length direction, and specifies an area of the same size and position in the entire image as when the first fitting function was created as the search area. The size of the search area is the same as the size of the search area when the first fitting function is created.

The control unit 20 designates an area in the reference image that is displaced by pixels from the position of the search area along the baseline length direction as a comparison area, with one of the base image and the reference image displaced by 0.5 pixels in the baseline length direction. The control unit 20 calculates the degree of difference between the partial images included in the search area and the comparison area, with one of the base image and the reference image displaced by 0.5 pixels in the baseline length direction.

The control unit 20 creates a second fitting function based on the degree of dissimilarity calculated for each of the above-mentioned multiple comparison regions while one of the standard image and the reference image is displaced by 0.5 pixels in the baseline length direction. The second fitting function is a function that indicates the relationship between the degree of dissimilarity and the amount of movement of the comparison region in the baseline length direction relative to the search region.

The control unit 20 may create at least one of a second coarse function and a second detailed function as the second fitting function. The second coarse function may be a function in which two dissimilarity degrees whose movement amounts are adjacent to each other are connected by a line segment. The second detailed function is created by performing approximate straight line fitting or parabolic fitting so that the function passes through the dissimilarity degree corresponding to the aforementioned movement amount.

The control unit 20 calculates the disparity based on the created first fitting function. More specifically, the control unit 20 may calculate the amount of movement that minimizes the degree of difference of the first fitting function as the disparity. The first fitting function for calculating the disparity may be either the first coarse function or the first detailed function. In this embodiment, the first fitting function for calculating the disparity is the first detailed function.

In a configuration in which a second fitting function is further created, the control unit 20 may calculate the disparity based on the created first fitting function and second fitting function. More specifically, the control unit 20 may calculate, as the disparity, the movement amount at which the disparity is minimum in a fitting function obtained by summing or averaging the disparities at the same movement amount between the first fitting function and the second fitting function.

The control unit 20 also calculates the disparity using dynamic programming, in other words, the SGM (Semi-Global Matching) method. The SGM method is briefly described below. In the SGM method, the amount of pixel shift with respect to the minimum value of the synthesis cost S(p, d) defined by equation (1) is calculated as the disparity.

In equation (1), p is the pixel of interest, d is the shift amount, and Lr(p, d) is the path cost value in the r direction for the shift amount d of the pixel of interest p, as determined by equation (2).

In equation (2), C(p, d) is the cost for a shift amount of d, r is the pixel one pixel before the pixel of interest p in the r direction, Lr(p-r, d) is the path cost value in the r direction for a shift amount d of the pixel one pixel before the pixel of interest p on the path in the r direction, i is a variable, and _P1 and _P2 are penalty values for which _P2 > _P1 holds.

The control unit 20 selects one of the disparities calculated based on the first fitting function and the disparity calculated using dynamic programming as the disparity of the subject image in the search area based on at least one of the extreme value of the first fitting function and the slope near the extreme value. The control unit 20 may further select one of the two disparities based on at least one of the extreme value of the second fitting function and the slope near the extreme value.

First, the selection of disparity based on at least one of the extreme value of the first fitting function and the slope near the extreme value (hereinafter also referred to as "selection of disparity based on the first fitting function") will be described below.

In this embodiment, the control unit 20 may select the disparity calculated based on the first fitting function when the minimum value of the first fitting function is equal to or greater than the first minimum threshold and the slope near the minimum value is equal to or greater than the first slope threshold. The slope near the minimum value is, for example, the maximum value among the absolute values of the slope between the movement amount that is the minimum value and the integer movement amount adjacent to the movement amount in the positive direction and the slope between the integer movement amount adjacent to the movement amount in the negative direction. The control unit 20 selects the disparity calculated using dynamic programming when the minimum value of the first fitting function is equal to or less than the first minimum threshold and the slope near the minimum value is equal to or less than the first slope threshold.

The first fitting function for selecting the disparity may be either a first coarse function or a first detailed function. In this embodiment, the first fitting function for selecting the disparity is a first coarse function.

Furthermore, in this embodiment, the control unit 20 may select the disparity calculated based on the first fitting function when either the minimum value of the first fitting function is equal to or greater than the first minimum threshold, or the slope near the minimum value is equal to or greater than the first slope threshold. In other words, the control unit 20 may select the disparity calculated based on the first fitting function when the minimum value of the first fitting function is equal to or less than the first minimum threshold and the slope near the minimum value is not equal to or less than the first slope threshold.

Alternatively, in this embodiment, the control unit 20 may select the disparity calculated using dynamic programming if either the minimum value of the first fitting function is equal to or less than the second minimum threshold and the slope near the minimum value is equal to or less than the second slope threshold. The second minimum threshold is less than the first minimum threshold. The second slope threshold is less than the first slope threshold.

Next, the selection of disparity based on at least one of the extreme value of the first fitting function and the slope near the extreme value, and at least one of the extreme value of the second fitting function and the slope near the extreme value (hereinafter, also referred to as "selection of disparity based on the first fitting function and the second fitting function") will be described below.

In this embodiment, the control unit 20 may select a disparity by using the maximum value among the minimum values of the first fitting function and the second fitting function (hereinafter also referred to as the "largest minimum value") instead of the minimum value of the first fitting function in selecting a disparity based on the first fitting function. The control unit 20 may select a disparity by using the maximum value among the slopes of the first fitting function and the second fitting function (hereinafter also referred to as the "maximum slope") instead of the slope near the minimum value of the first fitting function in selecting a disparity based on the first fitting function.

More specifically, when the maximum minimum value is equal to or greater than the first minimum threshold and the maximum slope is equal to or greater than the first slope threshold, the control unit 20 selects the disparity calculated based on the first fitting function and the second fitting function as the disparity of the subject image in the search region. When the maximum minimum value is equal to or less than the first minimum threshold and the maximum slope is equal to or less than the first slope threshold, the control unit 20 selects the disparity calculated using dynamic programming.

Furthermore, the control unit 20 may select the disparity calculated based on the first fitting function and the second fitting function when either the maximum minimum value is equal to or greater than the first minimum threshold value and the maximum slope is equal to or greater than the first slope threshold value. Alternatively, the control unit 20 may select the disparity calculated using dynamic programming when either the maximum minimum value is equal to or less than the second minimum threshold value and the maximum slope is equal to or less than the second slope threshold value.

The control unit 20 may calculate the position in the world coordinate system of the subject corresponding to the subject image in the search area based on the selected parallax, exterior orientation parameters, and interior orientation parameters. The control unit 20 may control the output unit 18 to output the calculated position in the world coordinate system of the subject to the external device 14. The control unit 20 may sequentially designate a part of the reference image or the entire reference image as a comparison area and calculate the position of the subject for each pixel.

Next, the position calculation process executed by the control unit 20 in this embodiment will be described with reference to the flowchart in FIG. 4. The position calculation process starts each time one frame of the base image and the reference image is acquired from the stereo camera 12.

In step S100, the control unit 20 specifies a search area from among unspecified areas that should be specified for the base image. The control unit 20 also specifies multiple comparison areas for the reference image. After the specification, the process proceeds to step S101.

In step S101, the control unit 20 creates a first fitting function and a second fitting function based on the degree of difference between the search area specified in step S100 and each of the multiple comparison areas. After the functions are created, the process proceeds to step S102.

In step S102, the control unit 20 calculates the disparity based on the first fitting function and the second fitting function. After the calculation, the process proceeds to step S103.

In step S103, the control unit 20 calculates the disparity using dynamic programming. After the calculation, the process proceeds to step S104.

In step S104, the control unit 20 selects the minimum values of each of the first fitting function and the second fitting function. After the selection, the process proceeds to step S105.

In step S105, the control unit 20 calculates the slopes near the minimum values of the first fitting function and the second fitting function selected in step S104. After the calculation, the process proceeds to step S106.

In step S106, the control unit 20 selects the maximum minimum value among the minimum values of each of the first fitting function and the second fitting function selected in step S104. The control unit 20 also selects the maximum slope among the slopes near the minimum values of each of the first fitting function and the second fitting function calculated in step S105. After the selection, the process proceeds to step S107.

In step S107, the control unit 20 determines whether the maximum minimum value selected in step S106 is less than or equal to the first minimum threshold and the maximum slope selected in step S106 is less than or equal to the first slope threshold. If the maximum minimum threshold is less than or equal to the first minimum threshold and the maximum slope is less than or equal to the first slope threshold, the process proceeds to step S108. If the maximum minimum threshold is not less than or equal to the first minimum threshold and the maximum slope is not less than or equal to the first slope threshold, the process proceeds to step S109.

In step S108, the control unit 20 selects the disparity calculated using dynamic programming in step S103 as the disparity of the subject image in the search area specified in step S100. After the selection, the process proceeds to step S110.

In step S109, the control unit 20 selects the disparity calculated based on the first fitting function and the second fitting function calculated in step S102 as the disparity of the subject image in the search area specified in step S100. After the selection, the process proceeds to step S110.

In step S110, the control unit 20 calculates the position of the subject corresponding to the subject image included in the search area specified in step S100 based on the parallax, exterior orientation parameters, and interior orientation parameters selected in one of steps S108 and S109. After the calculation, the process proceeds to step S111.

In step S111, the control unit 20 controls the output unit 18 to output the position calculated in step S110 to the external device 14. After the output, the process proceeds to step S112.

In step S112, the control unit 20 determines whether all of the areas to be specified in the reference image have been specified as search areas. If some areas have not been specified, the process returns to step S100. If all of the areas to be specified have been specified, the position calculation process ends.

The image processing device 10 of this embodiment configured as described above creates a first fitting function that indicates the relationship between the disparity between the search area and the comparison area relative to the amount of movement of the comparison area in the baseline length direction based on multiple disparities between the search area in the standard image and the comparison area displaced in pixel units along the baseline length direction of the stereo camera 12 in the reference image, and selects either the disparity calculated based on the first fitting function or the disparity calculated using dynamic programming as the disparity of the subject image in the search area based on at least one of the extreme value of the first fitting function and the slope near the extreme value. Generally, when the search area and the area near the search area are similar, such as in a textureless area, the difference between the disparities of each of the multiple comparison areas near the extreme value is small. A small difference in disparity indicates that the visual difference between the multiple comparison areas is extremely small. Therefore, in this case, the accuracy of the disparity calculated by block matching using the fitting function decreases. On the other hand, even if the search area and the area near the search area are similar, the accuracy of the disparity calculated using dynamic programming is higher than the accuracy of the disparity calculated by flock matching. Therefore, the image processing device 10 having the above-mentioned configuration can select a more accurate disparity based on at least one of the extreme value and the slope near the extreme value of the first fitting function, which can change depending on the visual difference between the search area and the area near the search area. Therefore, the image processing device 10 improves the accuracy of the disparity used to calculate the position of the subject.

In addition, in the image processing device 10 of this embodiment, the disparity increases as the visual difference between the search area and the comparison area increases, and the control unit 20 selects the disparity calculated based on the first fitting function when the minimum value of the first fitting function is equal to or greater than the first minimum threshold and the slope near the minimum value is equal to or greater than the first slope threshold, and selects the disparity calculated using dynamic programming when the minimum value is equal to or less than the first minimum threshold and the slope is equal to or less than the first slope threshold. In general, when there are few features in the search area, the disparity between the search area and the comparison area is relatively small. In addition, when there are few features in the search area, the change in disparity relative to the amount of movement, in other words the slope, is relatively small. Therefore, the image processing device 10 having the above-mentioned configuration can select the disparity calculated by block matching for a search area that is likely to have many features, and can select the disparity calculated using dynamic programming for a search area that is likely to have few features.

In addition, the image processing device 10 of this embodiment selects the disparity calculated based on the first fitting function when either the minimum value is equal to or greater than the first minimum threshold and the slope is equal to or greater than the first slope threshold. With this configuration, the image processing device 10 can select the disparity calculated by block matching when the minimum value is equal to or less than the first minimum threshold and the slope is not equal to or less than the first slope threshold. Therefore, the image processing device 10 can select the disparity calculated by block matching except when there is a high possibility that the accuracy of the disparity calculated by block matching is reduced. As a result, the image processing device 10 can select the disparity calculated by dynamic programming only for a search area where the accuracy of the disparity calculated by dynamic programming is likely to be higher than that of the disparity calculated by block matching. Therefore, the image processing device 10 further improves the accuracy of the disparity used to calculate the position of the subject.

In addition, the image processing device 10 of this embodiment creates a second fitting function that indicates a relationship between the degree of difference between the search area and the comparison area with respect to the amount of movement of the comparison area in the base line length direction based on a plurality of degrees of difference between the search area of the base line image and the comparison area that is moved in the base line length direction in the reference image in a state where one of the base image and the reference image is displaced by 0.5 pixels in the base line length direction, and selects the parallax of the subject image in the search area based on at least one of the extreme value of the second fitting function and the gradient near the extreme value. Since the first fitting function is created based on the degree of difference with respect to the amount of movement in pixel units, as shown in FIG. 5, a shift Δi (hereinafter, sometimes referred to as a "shift amount shift") may occur between the amount of movement i _min corresponding to the extreme value Ev _min of the fitting function f _min in which the unit amount of movement is infinitely small, and the amount of movement i corresponding to the extreme value Ev1 of the first fitting function f1. The shift amount deviation Δi is generally of a certain magnitude, and the extreme value Ev1 of the first fitting function f1 may indicate that there is a slight visual difference between the comparison area and the search area at the shift amount i corresponding to the extreme value Ev1. However, as shown in FIG. 6, when the shift amount deviation Δi is small, the extreme value Ev1 of the first fitting function f1 may be of the same magnitude as the extreme value _Evtl of the fitting function _ftl when the search area and the area near the search area are similar, indicating that the visual difference between the comparison area and the search area at the shift amount i corresponding to the extreme value Ev1 is extremely small. Therefore, when the shift amount deviation Δi is small, the accuracy of the selection of the disparity is improved by selecting using the extreme value of another fitting function with a larger shift amount deviation than the extreme value of the first fitting function f1. In general, as shown in FIG. 7, the shift amount deviation Δi of one of the shift amount i corresponding to the extreme value Ev1 of the first fitting function f1 and the shift amount i+0.5 or i-0.5 corresponding to the extreme value Ev2 of the second fitting function f2 may be larger than the other. Therefore, the image processing device 10 having the above-mentioned configuration can select at least one of the disparity calculated based on the first fitting function and the disparity calculated using dynamic programming based on the extreme values Ev1 and Ev2 of the first fitting function f1 and the second fitting function f2, so that the accuracy of the disparity can be further improved. Also, as shown in FIG. 5 and FIG. 6, when the shift amount deviation Δi becomes large, the deviation ΔEv between the extreme value Ev _min of the fitting function f _min in which the unit shift amount is infinitely small and the extreme value Ev1 of the first fitting function f1 (hereinafter, sometimes referred to as "extreme value deviation") may become large. Therefore, as shown in FIG. 5, when the extreme value deviation ΔEv becomes large, the gradient near the extreme value Ev1 may approach the gradient near the extreme value of the fitting function f _tl when the search area and the area near the search area are similar. Therefore, when the movement amount deviation Δi is large, the accuracy of the selection of the parallax is improved by selecting using the gradient of another fitting function that has a smaller extreme value deviation ΔEv than the gradient near the extreme value Ev1 of the first fitting function f1. In general, as shown in FIG. 7, either the extreme value Ev1 of the first fitting function f1 or the extreme value Ev2 of the second fitting function f2 may have a large extreme value deviation ΔEv. Therefore, the image processing device 10 having the above-mentioned configuration can select at least either the parallax calculated based on the first fitting function or the parallax calculated using dynamic programming based on the gradient near the extreme values Ev1 and Ev2 of the first fitting function f1 and the second fitting function f2, and the accuracy of the parallax can be further improved.

In addition, the image processing device 10 of this embodiment calculates, as the disparity, the movement amount that minimizes the disparity in a fitting function that sums or averages the disparities at the same movement amount between the first fitting function f1 and the second fitting function f2. With this configuration, in other words, the EEC method, the image processing device 10 can reduce systematic errors in subpixel estimation.

Although the embodiments of the present disclosure have been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications or corrections based on the present disclosure. Therefore, it should be noted that these modifications or corrections are included in the scope of the present disclosure. For example, the functions included in each component or each step can be rearranged so as not to cause logical inconsistencies, and multiple components or steps can be combined into one or divided. Although the embodiments of the present disclosure have been described mainly with respect to the device, the embodiments of the present disclosure can also be realized as a method including steps executed by each component of the device. The embodiments of the present disclosure can also be realized as a method, a program, or a storage medium on which a program is recorded, executed by a processor provided in the device. It should be understood that these are also included in the scope of the present disclosure.

For example, in this embodiment, the dissimilarity increases as the visual difference between the search area and the comparison area increases, but it may also decrease. In this configuration, the extreme value of the dissimilarity is a local maximum value. In this configuration, the local maximum value of the first fitting function is used to select the dissimilarity based on the first fitting function, and the comparison of the local maximum value with the threshold value is reversed. Also, in this configuration, the local maximum values of the first fitting function and the second fitting function are used to select the dissimilarity based on the first fitting function and the second fitting function, and the comparison of the local maximum value with the threshold value is reversed.

In this disclosure, descriptions such as "first" and "second" are identifiers for distinguishing the configuration. Configurations distinguished by descriptions such as "first" and "second" in this disclosure can have their numbers exchanged. For example, a first camera can exchange identifiers "first" and "second" with a second camera. The exchange of identifiers is performed simultaneously. The configurations remain distinguished even after the exchange of identifiers. Identifiers may be deleted. A configuration from which an identifier has been deleted is distinguished by a code. Descriptions of identifiers such as "first" and "second" in this disclosure should not be used solely to interpret the order of the configurations or to justify the existence of an identifier with a smaller number.

REFERENCE SIGNS LIST 10 Image processing device 11 Stereo camera device 12 Stereo camera 13 Moving object 14 External device 15 First camera 16 Second camera 17 Acquisition unit 18 Output unit 19 Memory 20 Control unit

Claims (9)

An acquisition unit that acquires a standard image and a reference image captured by a stereo camera;
a control unit that creates a first fitting function indicating a relationship between the degree of disparity of the search area and the comparison area with respect to an amount of movement of the comparison area in the baseline length direction, based on a plurality of degrees of disparity between a search area in the standard image and a comparison area that is displaced in the reference image along a baseline length direction of the stereo camera, and selects, based on at least one of an extreme value of the first fitting function and a slope near the extreme value, one of a disparity calculated based on the first fitting function and a disparity calculated using dynamic programming, as a disparity of a subject image in the search area. 2. The image processing device according to claim 1,
The dissimilarity increases as the difference between the search region and the comparison region increases,
the control unit selects the disparity calculated based on the first fitting function when the minimum value, which is the extreme value, is equal to or greater than a first minimum threshold and the slope is equal to or greater than a first slope threshold, and selects the disparity calculated using the dynamic programming when the minimum value is equal to or less than the first minimum threshold and the slope is equal to or less than the first slope threshold. 3. The image processing device according to claim 2,
The control unit selects the disparity calculated based on the first fitting function when either the minimum value is equal to or greater than the first minimum threshold and the slope is equal to or greater than the first slope threshold. 3. The image processing device according to claim 2,
The control unit selects the disparity calculated using the dynamic programming when either one of the following is satisfied: the minimum value is equal to or smaller than a second minimum threshold that is less than the first minimum threshold; and the tilt is equal to or smaller than a second tilt threshold that is less than the first tilt threshold. 5. The image processing device according to claim 1,
the control unit creates a second fitting function indicating a relationship between the degree of dissimilarity of the search area and the comparison area with respect to an amount of movement of the comparison area in the baseline length direction based on multiple degrees of dissimilarity between a search area of the standard image and a comparison area displaced pixel by pixel along the baseline length direction in the reference image with one of the standard image and the reference image displaced by 0.5 pixels in the baseline length direction, and selects a disparity of the subject image in the search area based on at least one of an extreme value of the second fitting function and a slope near the extreme value. 6. The image processing device according to claim 5,
The dissimilarity increases as the difference between the search region and the comparison region increases,
The control unit is
selecting a disparity calculated based on the first fitting function and the second fitting function as a disparity of a subject image within the search region when a maximum value among the minimum values of the first fitting function and the second fitting function is equal to or greater than a first minimum threshold and a maximum value among the slopes of the first fitting function and the second fitting function is equal to or greater than a first slope threshold;
when a maximum value among the minimum values of the first fitting function and the second fitting function is equal to or less than a first minimum threshold, and a maximum value among the slopes of the first fitting function and the second fitting function is equal to or less than a first slope threshold, the image processing device selects the disparity calculated using the dynamic programming as the disparity of the subject image within the search area. A stereo camera that captures a standard image and a reference image;
and an image processing device that creates a first fitting function indicating a relationship between the degree of disparity of the search area and the comparison area with respect to an amount of movement of the comparison area in the baseline length direction, based on a plurality of degrees of disparity between a search area in the standard image and a comparison area that is displaced in the reference image by a pixel unit along a baseline length direction of the stereo camera, and selects, based on at least one of an extreme value of the first fitting function and a slope near the extreme value, one of a disparity calculated based on the first fitting function and a disparity calculated using dynamic programming, as a disparity of a subject image in the search area. a first fitting function that indicates a relationship between a degree of dissimilarity of the search area and the comparison area relative to an amount of movement of the comparison area in the baseline length direction based on a plurality of degrees of dissimilarity between a search area in the reference image and a comparison area that is displaced in the reference image by a pixel unit along a baseline length direction of the stereo camera, and a dissimilarity calculation method that calculates a dissimilarity of a subject image in the search area based on at least one of an extreme value of the first fitting function and a gradient near the extreme value, and a dissimilarity calculation method that calculates a dissimilarity of a subject image in the search area based on the extreme value of the first fitting function. A step of capturing a standard image and a reference image by a stereo camera;
creating a first fitting function that indicates a relationship between the degree of difference between the search area and the comparison area relative to an amount of movement of the comparison area in the baseline length direction, based on a plurality of degrees of difference between the search area in the standard image and a comparison area that is displaced in the reference image in units of pixels along a baseline length direction of the stereo camera;
and selecting, based on at least one of an extreme value of the first fitting function and a gradient near the extreme value, one of a disparity calculated based on the first fitting function and a disparity calculated using dynamic programming as a disparity of a subject image within the search area.

Images